At Toby we decide, field by field, what may reach a model before we build any feature that calls one. The questions we ask of our own design are the ones worth putting to any vendor whose AI you use with client work, so this piece sets them out.
What leaves your desk
When you use a chat assistant, four kinds of information can travel to the provider:
- The question you type. If it names the client or quotes a figure from their file, that goes too.
- Anything you paste or attach. Ask about one line of a payroll export and the whole export travels.
- What a connector reads. An assistant connected to your accounting file, email or drive reads what its permission allows, which is often far more than the question needs.
- Your conversation history. Many assistants keep past chats so you can return to them, and some draw on earlier messages when they answer later ones.
Where it goes
The model runs on the provider's servers, and the region it runs in can differ from the region where your prompts, the answers and the logs are stored. Providers also keep records for abuse monitoring, and on some services staff can read a conversation that has been flagged.
Terms differ by plan. Business plans tend to exclude your data from training and keep it for shorter periods. A personal account may use your chats to improve the model unless someone turns that off. The plan a staff member signed up to with a personal email address is often not the plan the firm assessed.
Why it matters for a practice
The Office of the Australian Information Commissioner's guidance on commercially available AI products recommends, as a matter of best practice, that organisations do not enter personal information, and particularly sensitive information, into publicly available generative AI tools.
If personal information goes to a provider overseas, the Australian Privacy Principles generally require reasonable steps to make sure the recipient handles it in line with the principles, and section 16C of the Privacy Act makes the business that sent it accountable for the recipient's breaches.
For registered tax and BAS agents, the Code of Professional Conduct in section 30-10 of the Tax Agent Services Act 2009 says: "Unless you have a legal duty to do so, you must not disclose any information relating to a client's affairs to a third party without your client's permission." Whether a prompt sent to a model provider is a disclosure to a third party is a question for your own adviser. The cautious course is to get the client's permission or keep identifying details out of the prompt.
Questions to put to a vendor
- Which fields leave our file when a feature calls a model? Ask for the list of fields, not a description.
- Where is the model hosted, and where are prompts, answers and logs stored? Ask for the region names.
- Is anything we send used to train or improve a model, and is that a term of the contract or a setting someone can change?
- How long are prompts and answers kept, and who can read them, including the provider's own staff?
- What does the connector read, and can it write anything back to our file?
- Are names and identifiers removed before a model sees the data, and how?
How we handle it at Toby
At Toby the figures in an answer are calculated by ordinary code that contains no AI. AI has two jobs: choosing which tool a question goes to, and explaining a result in plain language. Before anything reaches a model it is stripped back to what that explanation needs, such as an account code, a label, a figure or a section reference. Tax file numbers, bank details, salaries, staff names and raw organisation identifiers never reach AI. Where an explanation has to point back to a particular finding, the identifier is replaced with a scrambled reference that only our side can map back. Nothing we send to a model is used to train it.
We hold that boundary with checks in code. Every change we make runs through tests that fail if a new path could carry a name or a raw identifier towards a model call. The servers, the database and the logs run in the AWS Sydney region, and the models the assistant will use are bound to a register of models we have measured on Amazon Bedrock in Sydney. The assistant is not live yet: the research box and the free Xero health check use no AI at all. A Xero connection is read-only and writes nothing back to your file.
Next
Trained on your data: what the phrase means in a vendor's terms, and how to check whether it applies to you.
Sources
- Office of the Australian Information Commissioner, Guidance on privacy and the use of commercially available AI products.
- Office of the Australian Information Commissioner, Australian Privacy Principles guidelines, Chapter 8: APP 8 Cross-border disclosure of personal information.
- Tax Agent Services Act 2009 (Cth), section 30-10.