Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WriteAPIBlocking.WritePoint does not always return error if influx not reachable #235

Closed
jo-me opened this issue Mar 14, 2021 · 5 comments · Fixed by #239
Closed

WriteAPIBlocking.WritePoint does not always return error if influx not reachable #235

jo-me opened this issue Mar 14, 2021 · 5 comments · Fixed by #239
Milestone

Comments

@jo-me
Copy link

jo-me commented Mar 14, 2021

Hi,

I noticed that the WriteAPIBlocking gives inconsistent return values when InfluxDB is not reachable.
When testing out the app behavior when Influx is not running, I noticed that some write attempts yield errors but many run through as if they worked fine (err==nil).

In my case I'm writing concurrently from 3 separate go routines (about 3 points each per second) calling a function with the code below. According to issues/docs the WriteAPIBlocking should be thread-safe.

Any ideas what could be the issue?

The code in questions is based on the basic blocking write example

	        startTime := time.Now()
		err := influx.writeAPIBlocking.WritePoint(context.Background(), newPoints...)
		if err != nil {
			logger.Errorf("Error saving data to InfluxDB. Reason: %s", err.Error())
			return err
		}
		log.Infof("Saved %d %s point(s) in %d ms", len(newPoints), dataset.MeasurementName, time.Since(startTime).Milliseconds())
		return nil

Steps to reproduce:
List the minimal actions needed to reproduce the behavior.

  1. Turn off influx db (or use wrong url/port)
  2. Attempt to write points (from several go routines)
  3. err struct is sometimes nil indicating that everything was written OK

Expected behavior:
a not reachable influxdb should consistently produce errors on write

Actual behavior:
err is sometimes nil

Specifications:

  • Client Version: 2.2.2
  • InfluxDB Version: 2.0.4
  • Platform: Ubuntu 18.04 x64
@vlastahajek
Copy link
Contributor

@jo-me, thanks for using this library.
What you are experiencing is the retry behavior.
This client automatically retries write in case of a write error. You can set the log level to DebugLevel, to see detailed logging from retrying:

    client := influxdb2.NewClientWithOptions(serverURL, authToken, influxdb2.DefaultOptions().SetLogLevel(log.DebugLevel))

After a failed write, the batch is put to the retry queue. Retry strategy uses exponential retry time when a server doesn't send info about a suggested retry time. The first interval is 5s, then 25s, and finally 125s. The maximum number of retries is 3.

If another write comes during the retry wait time, the batch is put to the retry queue and no error is returned.
Which, thinking about this again, is indeed confusing. There should be an API-defined error.

@jo-me
Copy link
Author

jo-me commented Mar 15, 2021

The behavior you describe would fit to the non-blocking write operation.
I'm still using the blocking write API for now. Shouldn't that deliver the error without any implicit retry mechanisms?

@DaveHasteSpringer
Copy link
Contributor

I am also noticing this behaviour, and like @jo-me I would expect the blocking write API to return an error immediately instead of attempting an implicit deferred retry.

As a work-around for this, I am tempted to SetMaxRetries() to 0, although from a quick glance at the code I fear that a value of 0 here will be interpreted as no maximum limit on retries.

@DaveHasteSpringer
Copy link
Contributor

As a work-around for this, I am tempted to SetMaxRetries() to 0, although from a quick glance at the code I fear that a value of 0 here will be interpreted as no maximum limit on retries.

For information, I have created PR #237 which should allow this work-around to be used.

@vlastahajek
Copy link
Contributor

Thanks for your feedback, @jo-me , @DaveHasteSpringer . I've discussed this also internally and I will remove retry logic from the blocking write.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants