Food pantries face many challenges, including recruitment and training of staff/volunteers, communications with staff/volunteers and clients, providing client choice, and emergency preparedness. The investigators will develop, implement, and evaluate the Support Application for Food Pantries (SAFPAS), a mobile application to address these concerns under normal and emergency operations, and assess its impact on 20 Baltimore food pantries, and on the healthiness of foods received by 360 food pantry clients using a randomized controlled trial design. If successful, the tested and refined app will support local food assistance programs throughout the United States.

Overview and Study Design The overall goal of this application is to develop and test a mobile application to improve food pantry services in low-income areas of Baltimore, increase access to healthier foods to clients, and enhance emergency preparedness and response capacity.

The first part of the study will involve meeting with key stakeholders to develop the Support Application for Food Pantries (SAFPAS) and testing the app for usability. This work will take place in Baltimore and Detroit. The second part is a pilot study in which food pantries located in Baltimore will be randomly selected to receive the SAFPAS app (n=10) or serve as comparison pantries (n=10). The investigators will introduce different features of the app to food bank staff, food pantry managers and staff, and food pantry clients to understand challenges related to using the app for day-to-day tasks and to evaluate the healthiness of the foods clients receive as the main outcome. Finally, the investigators will demonstrate the completed app to additional stakeholders in Baltimore and Detroit to learn more about feasibility and how to improve it.

The findings from this R34 application will be used to support a full-scale clinical trial, which will test the SAFPAS app across multiple cities, and will assess its impact on pantries (i.e., change in food stocking, staff capacity, distribution of healthier foods) and pantry clients (i.e., obesity and dietary intake, and food security). Findings from this study will help us to: 1) produce a functional and acceptable mobile app; 2) provide preliminary data needed to plan a full-scale clinical trial, including demonstrating preliminary efficacy; 3) set appropriate standards for implementation and evaluation; 4) demonstrate the investigators ability to recruit and retain large numbers of food pantries and pantry clients; and 5) identify unexpected challenges of introducing a mobile application in a food pantry setting and help determine potential solutions. For details on these future research plans see the Future Clinical Trial Description document.

Study setting and Collaborations The study will primarily take place in low-income neighborhoods of Baltimore, Maryland. Baltimore has 225 food pantries located in churches, schools, etc., all of which receive most of the food supply from the Maryland Food Bank (MFB). The investigators main partner will be the MFB, specifically the Baltimore branch which is the central "hub" for both Western Maryland and Eastern Shore branches. The MFB distributes roughly 40 million meals to food pantries, soup kitchens, shelters, and faith-based organizations throughout Maryland. The research team has worked with the MFB for more than five years on projects involving food pantries, including Fresh Shelves Healthy Pantries.87 The research team will also partner with various city stakeholders including the Food Policy Director, the Baltimore Food Policy Initiative, the Baltimore City Health Department, and the Baltimore City Office of Emergency Management (see Letters of Support). Additional formative and feasibility research will also be conducted in Detroit, including with local food pantries and the Gleaners Food Bank.

The study will take place in 6 phases:

Phase 1: Formative Research As part of the proposed research, the investigators will conduct formative research to refine the SAFPAS app into an effective mobile app. The formative research will include in-depth interviews with multiple types of key stakeholders, including food pantry directors and volunteers (n=12), MFB leadership (n=4), food pantry clients (n=20), and representatives of key Baltimore agencies (n=6). These interviews will focus on understanding reactions to a prototype of the app, including key features, in particular to eChoice, capacity-building (staff/volunteer training), and emergency preparedness information components, on refining these features, adding features as needed, and incorporating the recommendations. In addition, the investigators will assess the potential demand for the app beyond Baltimore by conducting in-depth interviews about the app prototype in low-income areas of Detroit, among approximately 6 food pantry directors, 4 food bank leaders, and 20 food pantry clients.

Interviews will be recorded, transcribed and analyzed using qualitative analysis software (Atlas.ti 22). The research team will develop a coding scheme to apply to interview transcripts (i.e., labeling sections of text with a word or phrase that captures what the interviewee was talking about). The investigators will then use these codes to identify common themes that emerged across multiple interviews. Products of the formative research will center around identifying and refining the most useful components of the SAFPAS app.

Phase 2: SAFPAS App Development Results from the formative research phase will be incorporated into the development of the SAFPAS mobile app. The investigators will develop the app using Bootstrap 4.0, a multi-platform framework for building web-based apps.89 While web-based apps are slightly less user-friendly than "native" iPhone and Android apps, the apps are more versatile in that the app will run on all smartphones and laptops. Moreover, adjustments to the system can be more easily accommodated.

The SAFPAS app will provide food pantry managers a convenient and cost-effective tool to support pantry activities, and for staying connected with food pantry clients, the food bank, and with emergency services in the event of an emergency. SAFPAS will feature an attractive user interface, and have 13 main modules:

Module 1 (Opening the App): Opening the SAFPAS application for the first time will direct the user through a series of onboarding pages that describe the key features of the application and the benefits of using it. The user will then be directed to a main screen to choose whether the user is a food pantry client, food pantry manager/volunteer, or food bank or other city agency representative (e.g., emergency operations center (EOC) representative).

Module 2 (Registration and 'Turn on Notifications'): All users will be directed towards a registration page where the user can enter the user email and a password and general information (depending on role selected in Module 1, this could include: household size, title, contact information, preferred pantry). Once registration is complete, users will receive a pop-up screen to "Turn on Notifications?". This will allow push notifications regarding important communications from, for example, the food pantry managers to all clients regarding a change in pantry hours.

Module 3 (Food Pantry Client Main Screen): If the user selected the "Food Pantry Client" role, the user will be directed to the client main screen. Users will be able to select 'preferred' food pantries from either a list of available pantries or from a map. The main menu will include quick links and visuals to: preferred food pantry information (hours, location, available volunteer shifts), volunteer opportunities, eChoice ordering system, and the option to receive nutrition education messages through push notifications.

Module 4 (eChoice Ordering System): Clients will be able to use the app to create food orders using the eChoice system. While users cannot order specific foods, the user will be able to indicate to the food pantry the types of desired items (i.e., breakfast, lunch, or dinner items), diet preferences (low sodium, low cholesterol, plant-based, vegetarian, Kosher, Halal, etc.), and allergies. This system will also thereby provide data on client preferences and report these statistics to the food bank and other stakeholders.

Module 5 (Communications Center): All users will have access to a communication center where the user can message contacts at different levels (food banks <-> food pantries <-> clients). This will include direct and group messaging. Users can choose the addressee from a contact list or from a group drop down menu (i.e., all volunteers for a food pantry). Pantry managers will be able to push notifications to groups of people from the communications center. The communications center will also be used for the provision of nutrition education messaging for clients as a means of improving client food choices.

Module 6 (Food Pantry Directors/Staff Main Screen): If the user selected the "Food Pantry Director" role, the user will be directed to the food pantry director main menu. The main menu will include visuals and links to the following modules: Volunteer Management, eChoice Order Management, Communications Center, 'Our Pantry Dashboard' with pantry statistics, and Volunteer/Staff Training.

Module 7 (Volunteer Management): Food pantry directors and staff will be able to recruit and schedule volunteers through the app. The scheduling feature will include a calendar of shifts, open and filled slots, a contact list of active and inactive volunteers, and the status of volunteer training (who has completed, in process, or hasn't started required training). This module is integrated with the communications center, so directors can easily push notifications to all volunteers or individuals.

Module 8 (eChoice Order Management): Directors and volunteers/staff will use the eChoice order management tool to identify client preferences and restrictions, match with foods in stock, and then put together client orders and track distribution. This feature will help pantries better identify food needs within the client network and pantries will be able to use this information to guide choices from the food bank.

Module 9 (Our Pantry Dashboard): Pantry directors will be able to collect information from users through the app. Data will be automatically displayed and visualized via the "Our Pantry Dashboard", which will include client statistics, real-time emergency situation information, and volunteer management (i.e., number of trainings completed). This Dashboard will also collect user feedback on the eChoice system and barriers to success. These data can be exported to the food bank and other stakeholders, including local city agencies, in different report formats.

Module 10 (Training Provision and Tracking): Food pantry staff/volunteers will have access to online training via the SAFPAS app. Directors will be able to efficiently track and manage required and recommended training for volunteers and staff. Training modules include the use of the SAFPAS app, food pantry-specific rules and regulations, nutrition education for meeting client diet preferences, food safety/hygiene, emergency preparedness, and federal food assistance program applications. Users will receive a sharable certificate upon completion of the training.

Module 11 (Food Bank/EOC Main Screen): If the user selected "Food Bank/EOC representative" at the time of registration, the user will be directed to the food bank main menu. The main menu for food bank users will be data-oriented to provide quick access to statistics on all food pantry clients, managers, and staff/volunteers using the app. There will also be a communications center where the food bank can quickly send messages to food pantries (from a specific contact list or to all food pantry-related users).

Module 12 (App Administrator/Usage Statistics): On the backend of the app, the administrator will be able to access the app's overall analytics, including usage, active versus inactive users, time per page, demographics, feedback, number of direct versus group messages, number of push notifications, and number of users per pantry. The administrator will also have access to the app's settings and control features.

Module 13 (Emergency Operations -- Data Exportation and Sharing): Data from the SAFPAS app will be available for sharing with governmental agencies for use in tracking and monitoring emergency situations. During an emergency activation, the National Incident Management System (NIMS) best practices dictate unit control centers report NIMS strength to the EOC for accountability purposes. This typically takes the form of a report on available personnel, capabilities, and supplies. In similar fashion, the investigators suggest a system where food distribution centers can provide a daily "strength" report via a web portal to the EOC. The granularity of such a report could depend on local needs. For example, a food pantry could report the number of clients, food insecurity status, food preferences, number of spots available for new clients, etc., on a daily basis through the web interface. The data would be stored on a server managed by the EOC. At the EOC level, a Dashboard will show the real-time status at each participating food pantry.

SAFPAS Functionality and Usability Testing Once the SAFPAS mobile app has been developed, it will be tested for system functionality, stability, and usability.

Functionality and Stability: Each component of the application will be tested on its own and all together as a combined application to ensure that it performs as expected. Scenarios and checklists will be developed to 'test' the system on different devices and ensure consistency. Once basic system tests are complete, the investigators will subject the SAFPAS app to stress testing to ensure, under unfavorable conditions, that the app does not crash. Different user-error scenarios will be documented and simulated. Finally, speed and responsiveness of the system will be tested under different connectivity conditions (i.e., Wi-Fi, cellular, hotspot).

Usability: The investigators will use client-centered design methods to evaluate and grade the user-friendliness of the app interface, evaluate the 'learning curve' of new users, followed by user proficiency testing on days 0, 1, and 3 after training - particularly for food pantry staff. The investigators will also plan repeated user-satisfaction assessments at various time points after deployment to assess whether the system meets the expectations of those who will be using it. SAFPAS will be configured to produce multiple outputs, including the capability to collect and store data, in a central server. The investigators will also build a simple survey within the app with rating scales and a text box so that users can provide feedback on any aspect of the program. A Structured Query Language (SQL) database will be used so that the data can be readily accessed using standard data analysis software such as Stata 14.1. This program-wide information will be continuously updated to keep the users and the research coordinators informed about the program operational status. Security issues are considered in the Data Safety and Monitoring Plan.

Phase 3: SAFPAS Pilot Evaluation Detailed sampling rationale, power calculations, eligibility, recruitment and retention information are provided in the PHS Human Subjects and Clinical Trials Information document. The investigators will assess the feasibility of the SAFPAS app from the perspective of food pantry clients, staff/directors, and the Maryland Food Bank (Aim 2). The investigators will evaluate the impact of the use of SAFPAS on client food procurement (Aim 3, primary focus) and food pantry stocking (availability, timeliness of delivery, quality, healthiness of foods offered), staff capacity (hygiene knowledge and behaviors), and distribution of healthy and unhealthy foods and beverages to clients in a pilot study employing a randomized controlled trial design of 20 pantries (Aim 4), and on a sample of 18 adult clients per pantry (n=360) to assess changes in relevant knowledge, self-efficacy, and intentions (Aim 5).

Using a list of Baltimore food pantries supplied by the MFB, the investigators will recruit pantries by contacting the food pantry directors and will maintain regular contact with the pantries and pantry staff throughout the study to achieve higher retention rates. The investigators will recruit food pantry clients who are referred by the participating pantry director/staff as a "usual client". Detailed recruitment and retention procedures are described in the Recruitment and Retention Plan, and detailed inclusion/exclusion criteria are presented in the Eligibility Criteria document. Informed consent will be obtained from all participants prior to data collection before the start of the study. Data will be collected at the start of the study and again at the end, and pantry-level data will be collected four times throughout the intervention. The investigators will also monitor the process of app implementation and use throughout the study (referred to as "process data"). Additional description of these instruments is provided in the Outcome Measures table.

Client-level instruments:

Food Assessment Scoring Tool (FAST) (Primary outcome): FAST was developed by Dr. Caitlin Caspi and her team at the University of Minnesota.93 This data collection form consists of 13 categories of food and beverages and several sub-categories within each. The total weight of the food type in a client bag is measured and used to calculate an overall score, ranging from 0-100, which also takes into account the healthfulness of each food category. While this tool was originally developed to measure overall healthiness of foods offered at pantries by assessing foods on pantry shelves, it can also be used to measure the healthfulness of individual client bags. In the investigators previous pilot trial with food pantries, Fresh Shelves, Healthy Pantries, the investigators found statistically significant improvements in FAST scores of pantry client bags in intervention pantries compared to the control pantries.87 The Client Impact Questionnaire (CIQ) will be a modified version of a survey that the investigators have used in previous studies with low-income Baltimore residents. This instrument will collect information including age, sex, income, food assistance program participation, food pantry usage, psychosocial measures (i.e., knowledge, self-efficacy, and intentions), and food-related behavior. In addition, the investigators will measure each participant's height, weight, and body composition (% body fat versus lean tissue. The CIQ will be done at the beginning and end of the study (two times) on the same sample of clients. All client measures will take place at the Johns Hopkins Bloomberg School of Public Health in a private office, or at the food pantry in a private location.

A questionnaire called the Adult Block Food Frequency Questionnaire 2014 (FFQ) that asks how often the client consumed 127 different foods over the past 30 days will be used to estimate client food and nutrient consumption (e.g., total energy intake, total fat, added sugar, sugar-sweetened beverage, and fruit and vegetable intake). Completed FFQs will be analyzed by the NutritionQuest company (Berkeley, California), and estimates of food and nutrient intakes will be generated. The FFQ will be done at the beginning and end of the study (two times).

Food Pantry-level instruments:

Food Pantry Environmental Checklist (FPEC): The FPEC is used to assess the availability of healthy and unhealthy foods and beverages at food pantries. Foods and beverages are categorized based on the United States Department of Agriculture's (USDA) five groups: dairy, fruits, vegetables, grains, and proteins. Additionally, there are two more categories for high-sugar and high-fat foods and beverages. Along with food, the FPEC has a section on the food pantries' policies on procuring, stocking, and distributing food items, including questions regarding the number of freezers/refrigerators available and which specific programs donate food, amongst others.87 Food Pantry Staff/Volunteer Impact Questionnaire (FPSIQ): The FPSIQ will be newly developed to assess food pantry staff and volunteer's knowledge and behaviors related to nutrition, food hygiene, and emergency management. Questions will correspond to the information presented in the SAFPAS training modules and will be done at the beginning and end of the study.

Data generated by the SAFPAS application:

The SAFPAS app will generate user statistics, including information on how often and in what ways people are using the app. This information will be stored in a database and will also be updated daily in a web-based Dashboard with graphical displays sized for a laptop. Information that will be captured and stored by the system for later analysis includes time(s) and duration of sessions, frequency of use, and other technical data (battery charge, connectedness). In addition, select data will be accessible to users in a graphical form through the Dashboard function.

Process evaluation: A key use of the app-generated data will be to understand how many people the app reached, how much each person used the app, and whether the app was used in the ways the investigators originally envisioned for the SAFPAS intervention. The team will develop specific standards that will be measured through the SAFPAS app itself. Examples will include the number of clients and food pantries participating, the number of trainings completed by food pantry volunteers, and how well the app is used. The ability to meet set standards over time will be monitored, and the investigators will work to improve how well these standards are met as the trial proceeds.

All instruments will be electronic forms developed using a data collection program called REDCap, as the investigators have used for the investigators recently completed multicomponent intervention trial.92 See Outcome Measures and Statistical Design and Power Plan for more details.

Phase 4: SAFPAS Pilot App Implementation The investigators will pilot the SAFPAS app in 10 intervention food pantries over a 12-month period in East and West Baltimore. The intervention pilot will be conducted in 3 stages (4 months each), differentiated by the types of SAFPAS app features activated. Intervention pantry clients who are part of the evaluation sample will be encouraged to download the app and learn its key features after the initial round of data collection. During the first weeks of each stage, training of participating food pantry directors/staff and MFB staff will take place - focusing on use of any new features. Initial training will be followed up by proficiency testing where pantry staff will be requested to complete certain functions (e.g., register a new client, locate an order, send a message, etc.). With each subsequent stage, more app features will be made available to users, building on the features from previous stages.

Phase 5: Post-Intervention Data Collection All data collection done prior to the start of the study will be repeated in Phase 5 to assess change associated with the SAFPAS intervention. In addition, the investigators will assess exposure to the different components of the SAFPAS app among food pantry directors and clients - to assess which features are used effectively. This will include both quantitative and qualitative open-ended questions which will permit us to assess user experiences with the app in terms of satisfaction, including challenges and potential solutions. The investigators will also conduct in-depth interviews with the investigators 10 intervention food pantry directors, food bank partners, and a subsample of 20 food pantry clients to understand feasibility of the SAFPAS app. This will include questions on acceptability (economic, cultural), operability (challenges, ease in using the app and its functions), and whether the user would continue to use the app (perceived sustainability). In addition, the investigators will conduct a follow-up study in 6-8 food pantries in Detroit, where the investigators will demonstrate the SAFPAS app, and assess feasibility through in-depth interviews. A specific objective is to determine similarities and differences in how people use and rate the app in Detroit compared to Baltimore. Additional description of feasibility constructs is provided in the Statistical Design and Power document.

Phase 6: Data Analysis A detailed description of data analysis procedures and hypotheses to be tested in relation to each study aim is provided in the Statistical Design and Power document. The quantitative data analysis plan is a series of steps including visual review of completed interviews, data entry, checking for mistakes, and then several stages of data analysis. The statistical plan is to: 1) examine intervention effects as measured by a change variable, subtracting the pretest score/intake from posttest score/intake (exploratory analysis), and then 2) create linear regression equations to check whether the intervention is related to the changes in scores. The primary hypothesis is: Compared with comparison pantries, client bags in intervention pantries will show a greater improvement (larger positive change) in FAST scores from pre-intervention to post-intervention. The investigators will compare the average change in FAST scores of participant bags from the intervention pantries to those from the comparison pantries. The investigators expect that the SAFPAS intervention will result in an average improvement (positive mean change) in FAST scores that is 12 FAST points better than the change in the comparison group. Additional hypotheses associated with other outcomes and how analyzed are presented in the Statistical Design and Power document.

Plans for Dissemination of the SAFPAS app:

Public deployment and dissemination of the app: Following the successful completion of the pilot (this R34) and clinical trials (future R01), the SAFPAS app will be publicly deployed via Google Firebase and made available on the Apple Store and Google Marketplace. Advertisement for app will occur via large organizational websites such as FeedingAmerica.org and foodpantries.org. Food pantries will also be recruited to advertise the app on the pantry website. Social media websites like Twitter, Facebook, Instagram, and TikTok will also be used to disseminate information about the app.

App Usage Training: While the investigators anticipate the app to be intuitive to use, the investigators will provide in-app help and training through (1) email support, (2) informational popups for new users, and (3) a help page with frequently asked questions and a training video. An in-app feedback and support form will also allow users to contact the development team with any issues.

App maintenance, updates, and removal: The principal investigators will be responsible for all future support and maintenance of the app. The investigators anticipate that the app will exist indefinitely. While it is anticipated that the app will exist indefinitely, it may be deemed at some future point that this web-based app is no longer needed. The investigators will post a message to the homepage of the app at least 30 days prior to the archiving of the website. The investigators will also notify all registered users via email. After 30 days, any user data within the app and the entire Google Firebase account will be deactivated and removed.

Inclusion Criteria:

• Regular client of the food pantry (visit food pantry/order from one or more participating food pantries at least 1 time/month), for the past year, identified by food pantry personnel Adult (18 years or older)
• Willing and able to use a mobile app through a smart phone (or other device) for making electronic Choice (echoice) selections and engaging in other communications

Exclusion Criteria:

• Anticipate moving out of Baltimore in the next 12 months (for pilot study)
• Pregnant (due to changes in diet, weight and body composition)
• Medically or behaviorally at risk as indicated by the food pantry
• No access to a smart phone or other web-enabled device and Wi-Fi