Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.

Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.

The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.

These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.

  • finitebanjo@lemmy.world
    link
    fedilink
    English
    arrow-up
    0
    ·
    3 months ago

    We shouldn’t wait, it is already basically illegal to sample the works of others so we should just pull the plug now.

    • stoy@lemmy.zip
      link
      fedilink
      English
      arrow-up
      0
      ·
      3 months ago

      The issue with legally pulling the plug is that it won’t stop AI baddies, only good AI companies who respect the law.

      The knowledge and tools are still out there.

      But when the bouble bursts it will tank AI globally.

      • finitebanjo@lemmy.world
        link
        fedilink
        English
        arrow-up
        0
        ·
        3 months ago

        good AI companies who respect the law

        When those come around maybe we can rethink our stance, but for now we should stop the AI baddies.

        • stoy@lemmy.zip
          link
          fedilink
          English
          arrow-up
          0
          ·
          3 months ago

          Which will only be possible with good old fashioned bouble bursting as I said.

          • finitebanjo@lemmy.world
            link
            fedilink
            English
            arrow-up
            0
            ·
            3 months ago

            Nah we can start enforcing the laws as they exist. OpenAI is using works of others commercially without permission.

            We don’t have to wait.

            • stoy@lemmy.zip
              link
              fedilink
              English
              arrow-up
              0
              ·
              3 months ago

              As I noted, that only works with a limited set of AI companies.

              They need to be in the juristiction of whatever government that decide to enforce the laws, if not, there is very little that can be done.

              Then, besides needing to be in the right juristiction, the punnishment needs to be large enough that you can’t just budget it away.

              Then any country doing this will know that they are deliberately getting rid of an important sector, while other countries will continue running their sectors.