Thank you for sharing this situation. Evaluating custom software development companies can indeed be tricky, especially when you're dealing with a specialized industry like medical billing and payroll. Your approach of using an RFI followed by a detailed RFP seems solid, as it allows you to assess a company’s methodology and experience before diving into specifics. Here are a few additional suggestions that might help refine your process:

1. Initial Screening: Before diving into in-depth RFPs, you might consider conducting a round of preliminary interviews or a "soft" RFI that helps you narrow down the list of potential candidates. In these interviews, focus on understanding each company's domain expertise, experience with similar projects, and their approach to solving complex business rules.

2. Look for Case Studies or Proof of Concepts: When asking companies to provide a white paper, also request detailed case studies of similar projects they've worked on, specifically in the healthcare or finance space. This will give you a sense of how they handle complex requirements and customization.

3. Focus on Agile Development: Since your client needs a tailored solution, it's crucial that the company has strong experience with agile methodologies. Make sure they can demonstrate how they adapt to changing requirements and ensure regular communication and feedback loops.

4. Cultural Fit and Collaboration: As part of your evaluation, pay close attention to how companies approach client collaboration. The best partners for a complex project will be those that view themselves as an extension of your team, not just as service providers. Perhaps conduct mock discovery sessions or workshops to see how they engage with your team and gather requirements.

5. References and Demos: As you mentioned, speaking with references and seeing live demos of their work are essential. When checking references, ask very specific questions about how the company handled challenges, changes, and the delivery process for previous clients in similar industries.

6. Evaluating Long-Term Viability: Since this is a custom solution, consider the long-term partnership potential. Evaluate the company's financial stability, growth trajectory, and their ongoing support and maintenance processes. This is important to ensure they can continue to support and enhance the software over time.

For a comprehensive review of development companies and their services, you might find this resource helpful: https://www.cleveroad.com/blog/software-development-companies-vancouver/.

Best of luck with your project!