u/pearlhoneytar

I’m looking for a sanity check about an AI chatbot app for NDIS participants.

It appears to be currently only available to NDIS participants. It is marketed to people aged 13+ with cognitive disabilities, psychosocial disabilities, processing difficulties, autism, acquired brain injury and/or intellectual disability.

The app is presented as helping with thinking, memory, understanding information, routines, decision making, daily life and NDIS goals. The website also uses a lot of language about building independence, consistent support, clinical frameworks, being “NDIS Designed” and to align with goals.

The founder/creator of the app also runs an NDIS service provider.

My main question is about the terms. The website could give the impression that this app is a tool to help someone think. But in the terms, they say the AI outputs may contain inaccuracies, omissions or bias, are not guaranteed to be accurate, complete, reliable, relevant or suitable, and that users should independently verify information before relying on it.

How does that work when the target users may include children and people whose disabilities may affect comprehension, judgement, memory, decision making or even their ability to identify mistakes?

Does that term negate the impression the website makes?

The website also appears to make strong claims about safety, cognitive support, NDIS goal alignment, independence, decision-making support and reducing incidents in SIL. I cannot see clear public evidence, independent testing, trials, clinical validation or external review supporting those claims. That may exist privately, but I cannot find it publicly.

There also seem to be timing questions. Some white paper style materials appear to be dated a year before the public launch and before the current company/operator appears to have been publicly registered. I’m not saying that is wrong, but it makes me wonder what evidence, testing, pilots or review existed when those claims were being made.

Also, does involvement and consultation with people with disabilities mean external consultation or does it include information from clients who use your provider service and/or users?

There is also public information that appears to discuss the app overseas as a commercial/scalable caretech product before months before launch, but the current marketing says it is designed specifically for NDIS participants and NDIS plans.

Again, I’m not saying that is wrong, but it raises questions for me about what it claims to do, safeguards, funding pathways, conflicts of interest and whether users understand the possible commercial context of the app?

This is not an allegation, but if a product has commercial intentions from inception, launches on NDIS, and is possibly using NDIS participants as their like, first wave of release, and using all that information and data to develop the product, is their a responsibility for the consumer/NDIS participants to be made aware of this?

I think accessible technology is a good thing. I’m just trying to understand what standard practice is here.

Would people with experience see this as a legitimate concern or am I reading too much into it?

I’m looking for a sanity check about an AI chatbot app for NDIS participants.

It appears to be currently only available to NDIS participants. It is marketed to people aged 13+ with cognitive disabilities, psychosocial disabilities, processing difficulties, autism, acquired brain injury and/or intellectual disability.

The app is presented as helping with thinking, memory, understanding information, routines, decision making, daily life and NDIS goals. The website also uses a lot of language about building independence, consistent support, clinical frameworks, being “NDIS Designed” and to align with goals.

The founder/creator of the app also runs an NDIS service provider.

My main question is about the terms. The website could give the impression that this app is a tool to help someone think. But in the terms, they say the AI outputs may contain inaccuracies, omissions or bias, are not guaranteed to be accurate, complete, reliable, relevant or suitable, and that users should independently verify information before relying on it.

How does that work when the target users may include children and people whose disabilities may affect comprehension, judgement, memory, decision making or even their ability to identify mistakes?

Does that term negate the impression the website makes?

The website also appears to make strong claims about safety, cognitive support, NDIS goal alignment, independence, decision-making support and reducing incidents in SIL. I cannot see clear public evidence, independent testing, trials, clinical validation or external review supporting those claims. That may exist privately, but I cannot find it publicly.

There also seem to be timing questions. Some white paper style materials appear to be dated a year before the public launch and before the current company/operator appears to have been publicly registered. I’m not saying that is wrong, but it makes me wonder what evidence, testing, pilots or review existed when those claims were being made.

Also, does involvement and consultation with people with disabilities mean external consultation or does it include information from clients who use your provider service and/or users?

There is also public information that appears to discuss the app overseas as a commercial/scalable caretech product before months before launch, but the current marketing says it is designed specifically for NDIS participants and NDIS plans.

Again, I’m not saying that is wrong, but it raises questions for me about what it claims to do, safeguards, funding pathways, conflicts of interest and whether users understand the possible commercial context of the app?

This is not an allegation, but if a product has commercial intentions from inception, launches on NDIS, and is possibly using NDIS participants as their like, first wave of release, and using all that information and data to develop the product, is their a responsibility for the consumer/NDIS participants to be made aware of this?

I think accessible technology is a good thing. I’m just trying to understand what standard practice is here.

Would people with experience see this as a legitimate concern or am I reading too much into it?