Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle target model returning None in atkgen #678

Merged
merged 2 commits into from
May 14, 2024
Merged

Conversation

leondz
Copy link
Collaborator

@leondz leondz commented May 13, 2024

handle this exception:

probes.atkgen.Tox:   0%|                                                                                                                             | 0/10 [00:00<?, ?it/s]
Asking to truncate to maxlength but no maximum length is provided and the model has no predefined maximum length. Default to no truncation. | 1/10 [00:06<00:56,  6.27s/it]
Traceback (most recent call last):
  File "/opt/conda/bin/garak", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/garak/_main.py", line 9, in main
    cli.main(sys.argv[1:])
  File "/opt/conda/lib/python3.10/site-packages/garak/cli.py", line 486, in main
    command.probewise_run(generator, probe_names, evaluator, buff_names)
  File "/opt/conda/lib/python3.10/site-packages/garak/command.py", line 212, in probewise_run
    probewise_h.run(generator, probe_names, evaluator, buffs)
  File "/opt/conda/lib/python3.10/site-packages/garak/harnesses/probewise.py", line 106, in run
    h.run(model, [probe], detectors, evaluator, announce_probe=False)
  File "/opt/conda/lib/python3.10/site-packages/garak/harnesses/base.py", line 93, in run
    attempt_results = probe.probe(model)
  File "/opt/conda/lib/python3.10/site-packages/garak/probes/atkgen.py", line 139, in probe
    response = generator.generate(challenge)[0].strip()
IndexError: list index out of range

@leondz leondz requested a review from jmartin-tech May 13, 2024 09:06
@leondz leondz added bug Something isn't working probes Content & activity of LLM probes labels May 13, 2024
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I think the generator may need some adjustment too, I suspect this can be handled more explicitly by just validated the result returned.

garak/probes/atkgen.py Outdated Show resolved Hide resolved
@leondz leondz merged commit dc4a1c1 into main May 14, 2024
6 checks passed
@leondz leondz deleted the bugfix/atkgen_noresp_guard branch May 14, 2024 05:55
@github-actions github-actions bot locked and limited conversation to collaborators May 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working probes Content & activity of LLM probes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants