The development of AONES was funded through a Locally Driven Collaborative Project grant from Public Health Ontario. One of the broader goals of the project was to evaluate the process of developing an AI-based-tool in a collaborative public health setting. To evaluate this process, all collaborators were invited to participate in a brief survey and an in-depth facilitated reflective process was conducted with the primary development team. The evaluation served to assess the strengths, opportunities and challenges and to share learnings more widely. The key findings are summarized below:

• Plan to be Flexible. Unlike traditional epidemiological studies, AI-based methods cannot always be firmly established in advance, especially when related technologies are in a state of rapid advancement. It is important to define project scope with clear goals, but expect that methods may need to be revised as the project moves forward.
• Invest in Infrastructure. Carefully assess the IT infrastructure needs of the project in comparison to the current capacities and plan the changes that are needed with consideration of organizational policies and procedures (e.g., security, privacy, and confidentiality). Allot extra time at the start of the project to purchase and install infrastructure, as well as account for inevitable delays.
• Balance Custom vs Out-of-the-Box Tools. Consider the appropriate balance of custom development versus purchasing of tools (i.e., via subscription models) for specific sub-tasks to ensure project time is best allocated to the new and specific elements of the project.
• Lower Your Expectations for Collaborative Development. External collaboration in the development phases is a difficult task due to both technical and organizational limitations. Having a strong and experienced internal team may be needed first to make a smoother collaboration with external partners. Navigating external organizations’ protocols and procedures can also add a layer of complexity to the project that may slow down progress.
• Engage Collaborators Early (and Late). Capitalize on the expertise and creativity of collaborators and potential tool users. Engage the team early in the brainstorming/idea-generation and scoping phases to ensure the project targets and addresses a relevant real-world problem. Plan feedback sessions to help guide the development and implementation of any data products and tools from the project.
• Plan for Built-in Sustainability. Applied AI projects require continual maintenance and may have ongoing costs (e.g., cloud computing, subscription services). For the project to be sustainable, it requires staff members with adequate knowledge and expertise in applied AI projects to be available to support any issues on an ongoing basis. Changes to the data sources, packages within the project or subscription services may require significant future investment to keep the tools operational. Consider budgeting for a short-term or long-term sustainability plan.
• Share the Knowledge. Collaboration and sharing of projects with other public health units can reduce duplication of work and share knowledge and practices. With advances in AI, there are also more resources and training available that make it easier to enter the field. However, do consider organizational policies on intellectual property before openly sharing methods or code.