Timestamp	What types of logging are available in your implementation of choice?	If you have implemented a structured logging format: why did you decide to do so? What are the key benefits?	If you have not implemented a structured logging format: why not? Do you plan to do so in the future?	Would you consider completely replacing your unstructured logging with purely structured logging? Why (not)?	How important are the following aspects of a structured logging format? [Small file size]	"How important are the following aspects of a structured logging format? [Easy integration in QUIC implementation (e.g., availability of libraries)]"	How important are the following aspects of a structured logging format? [Streamable (files do not need to be read/written in full)]	How important are the following aspects of a structured logging format? ['Grep'-able output (direct string search on a file)]	How important are the following aspects of a structured logging format? [(De)serialization performance]	How important are the following aspects of a structured logging format? [Ability to log raw packet (payload) data]	How important are the following aspects of a structured logging format? [Easy to load in (web-based) tooling]	Do you use (public) logs from other implementations?	How often do you use the following tools when debugging QUIC/H3 implementations? [Command-line output]	How often do you use the following tools/visualizations when working with QUIC/H3 implementations? [Texteditor / command line interface]	How often do you use the following tools/visualizations when working with QUIC/H3 implementations? [Wireshark]	How often do you use the following tools/visualizations when working with QUIC/H3 implementations? [qvis sequence diagram]	How often do you use the following tools/visualizations when working with QUIC/H3 implementations? [qvis congestion graph]	How often do you use the following tools/visualizations when working with QUIC/H3 implementations? [qvis multiplexing graph]	How often do you use the following tools/visualizations when working with QUIC/H3 implementations? [quictrace congestion graph]	"If you have used custom tools, or public tools not in the above list, please provide additional information here."	How often do you use the following tools/visualizations when working with QUIC/H3 implementations? [qvis statistics overview]	What tools (or features of tools) that currently do not exist yet would be most useful to you?	Do you have some examples of bugs or issues that additional tools helped you to identify more easily than if you had not used tools? Or issues that you would probably not have found without extra tooling?	How important is the availability of public/open-source tools to your choice of logging format?	"For which use cases do you envision using structured logging and extra tooling? [Teaching new people (students, interns, junior team members)]"	"For which use cases do you envision using structured logging and extra tooling? [Debugging newly implemented features (e.g., multipath, new application layer protocols)]"	"For which use cases do you envision using structured logging and extra tooling? [Debugging live deployment issues/bugs (e.g., problems with consumer traffic)]"	"For which use cases do you envision using structured logging and extra tooling? [Evaluation, testing and optimization of (live) deployments (e.g., A/B testing different configurations)]"	How important of a feature is structured logging (and subsequent tooling support) in your overall offering/strategy? 	Do you feel structured logging would be useful for non-QUIC/H3 setups? Why (not)?	Is there anything else you would like to share?	For which use cases do you envision using structured logging and extra tooling? [(Academic) research]	How important are the following aspects of a structured logging format? [Ability to easily log new custom event types/categories (flexibility)]	"How often do you use the following tools/visualizations when working with QUIC/H3 implementations? [Custom, in-house tooling]"	"If you have implemented a structured logging format (such as qlog), would you consider to (or already plan to) remove it at some point? Why (not)?"	"If you indeed use logs from other implementations, please provide some additional insight into when especially this is useful to you"	How important are the following aspects of a structured logging format? [Row 9]	Have you used other tools that are not in the 
3/17/2020 14:38:13	"qlog (structured), custom format (structured) (e.g., in-house event tracing framework)"	"Allows for decoding not only the QUIC information, but also payload as well, which matters to us for corellating data in the one place."			Of little importance	Of average importance	Of average importance	Of average importance	Of little importance	Very important	Very important	No or only rarely (I typically ask the other implementers to identify possible errors)		Very frequently	Occasionally	Rarely	Rarely	Rarely	Never	"Implementations can output new-line delimited JSON (so each line contains a discreet JSON blob). This can either be loaded into browser-based tools that can decode, or, more often than not, used in conjunction with jq, grep, etc."	Rarely			Important (we use outside tooling and/or adapt open-source projects to our needs)	Sometimes	Often	Sometimes	Often		"I already have used qvis for DoQ, any maybe being able to use it to better understand packet flow for other DNS protocols (like DoT/DoH, Do53) would be useful."		Sometimes	Very important	Occasionally	"I don't think it'll be removed, but rather its use will be made limited to only when we need to dive into protocol-level debugging where application logging and telemetry can't provide us information."			
3/17/2020 15:34:03	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), custom format (unstructured) (e.g., command line output)"	I just needed some quick and dirty internal stats that I can use bash tool to grep and process	"I mostly mingle with other peoples' implementation, if this is not supported I will not add it"	Not implementing QUIC just using it for research. For research: seems like a huge amount of work to replace it.	Of little importance	Very important	Very important	Absolutely essential	Very important	Of little importance	Very important			Frequently	Frequently	Never	Never	Never	Never		Never			Important (we use outside tooling and/or adapt open-source projects to our needs)	Almost always	Almost always	Sometimes	Often		"Yes, I've been debugging TCP for ages and seeing what qvis does makes me wish somebody had implemented it for TCP. "	"While I pointed out that file size is not important to me, I did so because I would compress these files (e.g., using brotli). Yet brotli is notoriously slow on the compression but blazing fast to decompress which would make it challenging to record large volumes of logs. Would it make sense to provide a compression scheme (e.g., like a zlib dictionary) given the typical fields for a certain protocol to speed up the compression?"	Often	Absolutely essential	Rarely				
3/17/2020 15:57:19	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), custom format (structured) (e.g., in-house event tracing framework)"	We use the REDACTED framework on Windows and REDACTED on Linux with our own custom events (not qlog). These are designed to be automatically processed by our own tools.			Absolutely essential	Absolutely essential	Of little importance	Absolutely essential	Absolutely essential	Of average importance	Of little importance	No or only rarely (I typically ask the other implementers to identify possible errors)		Very frequently	Rarely	Never	Never	Never	Never	Windows Performance Analyzer	Never			Neutral (it does not influence our choice of logging format)	Seldom	Often	Sometimes	Sometimes	Very important	"In Windows, we use ETW everywhere."	"Less reliance should be put on web based tooling. They cannot be as efficient and effective, IMO, as an actual installed application."	Seldom	Absolutely essential	Very frequently	"No. It's designed such that it can be compiled out if necessary, but we'd never remove it."			
3/17/2020 19:29:39	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), OMNeT++ result recording"		"We still consider implementing qlog, though it has not the highest priority. We hope to find a student who is willing to implement a qlog output."		Of average importance	Very important				Of little importance	Of little importance	No or only rarely (I typically ask the other implementers to identify possible errors)		Very frequently	Never	Never	Never	Never	Never	We use the ability of OMNeT++ to plot data for a first insight. We also used packetdrill to test static behavior of our implementation.	Never			Very important (we have implemented a type of logging only to be able to use outside tooling)	Sometimes	Often	Seldom	Seldom		"Yes, in my opinion, structured logging would be useful for every bigger software. However, I'm not sure if one logging format let me find every potential bug."		Almost always	Very important	Very frequently				
3/18/2020 6:25:34	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), qlog (structured), quictrace (structured), custom format (unstructured) (e.g., command line output)"	"1. Availability of tools (quic-trace and qvis, for example) 2. ability to write custom parsing tools to identify ""interesting"" traces from a huge pile of production traces"		"In production, I will only run structured logging. The unstructured logging will likely remain in place for a while, at least until qlog can record *all* the information that I'm outputting in unstructured form now."	Of average importance	Very important	Absolutely essential	Of little importance	Absolutely essential	Not important	Absolutely essential	"No or only rarely (I typically ask the other implementers to identify possible errors), Yes, but mainly unstructured logs"		Very frequently	Very frequently	Frequently	Frequently	Rarely	Occasionally		Frequently	"A tool that takes a bunch of traces and identifies ""interesting"" ones. For example, such a tool could parse all the traces, and order them by packet loss rate, retransmission rate, RTT, (scaled) number of PTO events, (scaled) number of BLOCKED frames sent / received, etc. There are a lot of interesting metrics that can be used to identify traces where things don't go smoothly, and which might warrant a closer look."	"The qvis sequence diagram is very helpful identifying problem with loss detection, especially during the handshake."	Very important (we have implemented a type of logging only to be able to use outside tooling)	Never	Almost always	Almost always	Almost always	Not having structured logging means flying blind. Having some actionable metrics is absolutely essential.	Not really. Most applications already have their own logging format.		Never	Absolutely essential	Rarely	Of course not. Having the code in place (but disabling it at runtime) has negligible overhead.			
3/18/2020 11:29:11	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE)"				Of average importance	Absolutely essential	Of average importance	Of average importance	Of little importance	Not important	Of little importance	"Yes, but mainly structured logs (e.g., in combination with qvis)"		Very frequently	Very frequently	Very frequently	Very frequently	Very frequently	Very frequently		Very frequently			Very important (we have implemented a type of logging only to be able to use outside tooling)	Almost always	Almost always	Almost always	Almost always				Almost always	Of little importance	Very frequently				
3/18/2020 12:19:28	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), qlog (structured), custom format (unstructured) (e.g., command line output)"			"Only if a realtime processing tool was available to output the json into human-readable format, to be dumped to the terminal."	Of average importance	Very important	Very important	Very important	Of average importance	Not important	Very important	"Yes, but mainly structured logs (e.g., in combination with qvis)"		Very frequently	Occasionally	Occasionally	Occasionally	Never	Never		Never	Offline (local) execution for qvis & the other q-tools. 	The qvis congestion graph is very useful.	Very important (we have implemented a type of logging only to be able to use outside tooling)	Never	Often	Often	Often				Sometimes	Of average importance	Very frequently		Insight into internals of the other stack		
3/19/2020 9:02:56	qlog (structured)	I felt that visualization by qvis would help me a lot			Of little importance	Very important	Absolutely essential	Very important	Of little importance	Of little importance	Of average importance	"Yes, both unstructured and structured logs"		Very frequently	Very frequently	Very frequently	Never	Never	Never		Never	"Error handling checker like ""h2spec"""		Important (we use outside tooling and/or adapt open-source projects to our needs)	Often	Often	Seldom	Seldom	Making it optional so that users can use it if they want to report bugs	"For HTTP/2, yes."		Sometimes	Of average importance	Never	No			
3/24/2020 12:42:23	"We define USDT probes, that can be used for logging in whatever format."				Of average importance	Of little importance	Very important	Of average importance	Of average importance	Of little importance	Very important	"Yes, but mainly structured logs (e.g., in combination with qvis)"		Very frequently	Rarely	Never	Never	Never	Occasionally		Never			Very important (we have implemented a type of logging only to be able to use outside tooling)	Almost always	Almost always	Almost always	Almost always	"It's a must-have. As an example, We use event logs (and probes) for CI."	"Maybe. Though, we do not care. As stated previously, we care about the probe points and what we expose through them. How to use them is up to each use-case. Some of our use-case might use qlog (when we are looking into QUIC transport performance). Or we might write into a SSLKEYLOGFILE using the same probe points. Or we would do a HTTP-level event logging in our own format."	"I highly appreciate your work in qlog and qvis. We would definitely be using them. IIUC, the only reason we haven't been using it so far is because we have not had enough time, and because existing toolset have provided what we need the most.
At the same time, as stated, we do not care much about the file format, because we use USDT probes. Our programs just define the probe points. It's up to each use-case to serialize the information gathered from those probe points. Our use of probe points is not limited to QUIC or something running on top of QUIC. We anticipate that we'd be using different serialization formats, depending on the use-case."	Almost always	Absolutely essential	Never	"We do not care about logging format, we only care about the probe points being defined."	Having logs of both sides is the often the best way to determine what is causing regression.		
3/24/2020 13:30:40	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE)"											"Yes, but mainly unstructured logs"			Frequently							Proper QUIC (stream) reassembly support in Wireshark which would enable full HTTP/3 support.	"Unencrypted PADDING outside the encrypted payload was not consistently handled between implementations (neqo does this, some other impl did not). quic-go used a different cipher than negotiated. Such wireformat issues could not be quickly identified in logs and benefited from a pcap with the raw messages + keys that can be interpreted."		Seldom	Often	Sometimes	Sometimes								"For interoperability, check whether the results are as expected or whether the implementation signalled an error."		
3/24/2020 17:01:42	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), custom format (unstructured) (e.g., command line output)"		We'd like to implement qlog but it's more a matter of time and effort to implement the feature.	I think keeping custom logging can allow us to debug some implementation-specific problems and quickly add temporary debug prints.	Of little importance	Very important	Absolutely essential	Not important	Of average importance	Very important	Very important	"Yes, both unstructured and structured logs"		Frequently	Rarely	Rarely	Rarely	Rarely	Rarely		Rarely			Neutral (it does not influence our choice of logging format)	Seldom	Often	Sometimes	Almost always				Sometimes	Very important	Never		When implementers are not available (different timezones makes it difficult sometimes).		
3/24/2020 18:35:23	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), custom format (structured) (e.g., in-house event tracing framework), custom format (unstructured) (e.g., command line output)"	"We favored a structured format because the systematically enforced uniformity makes it easier to manually analyze logs. Our format is custom because qlog didn't exist initially, and adhering to a standardized format takes extra effort, especially when using off-the-shelf general-purpose logging/tracing tools. We'd like to support qlog eventually, but it is not a high priority."			Not important	Very important	Of average importance	Very important	Of average importance	Of average importance	Of average importance	"Yes, both unstructured and structured logs"		Very frequently	Occasionally	Occasionally	Never	Never	Never		Never		"qvis logs from a third party implementation recently made it obvious that said implementation was erroneously failing to retransmit part of the handshake, saving us time and worry searching for a cause on our end."	Very important (we have implemented a type of logging only to be able to use outside tooling)	Often	Almost always	Almost always	Often	"It makes an excellent standard diagnostic tool, but we don't assign it any deeper significance than that."		I'm a big fan of the work you folks are doing! I expect you'll have a significant long-term benefit to the quality of QUIC implementations.	Often	Absolutely essential	Never	"No, as we believe it represents the state of the art of diagnostic tooling."	"Logs from third party implementations are immensely valuable for eliminating hypotheses and getting details on a failure that are unavailable on the protocol side, and allow for an extremely fast feedback loop compared to waiting for a response from a team that might be halfway across the world. Many of the bugs we've identified in our implementation and others' were found easily and quickly thanks to other implementations' logs being readily available."		
3/24/2020 18:40:39	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), custom format (unstructured) (e.g., command line output)"		"Structured logging is on our TODO list, but it has not made it to the top yet."	No. It is important to be able to process logs using standard command-line tools.	Of average importance	Very important	Of average importance	Of average importance	Of average importance	Of little importance	Very important	No or only rarely (I typically ask the other implementers to identify possible errors)		Very frequently	Never	Never	Never	Never	Never	"We have a few small scripts that process our logs.  For example, one graphs CWND."	Never			Neutral (it does not influence our choice of logging format)	Sometimes	Sometimes	Sometimes	Sometimes	This feature is not important commercially for us.			Never	Of average importance	Occasionally	"We won't remove our qlog implementation, for it will likely be useful."			
3/24/2020 19:43:32	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), qlog (structured), custom format (unstructured) (e.g., command line output)"	qlog is WIP but having tools to help visualize QUIC traffic behaviour is the main attraction.		"Based on implementing qlog, it is way easier to write ""printf(""foo %d"", bar)"" than having to think/worry about structured logging - for that reason I think there'll always be a need for unstructured
"	Of average importance	Very important	Absolutely essential	Very important	Of little importance	Of little importance	Of average importance	"Yes, but mainly unstructured logs, logs conunundrum: logs are useless until they are absolutely necessary. Typically I'd use tis type of log for interop failure diagnosis, so having a common format would allow easier parsing of other implementation logs"			Occasionally	Rarely	Occasionally	Occasionally	Never		Rarely	qpack analyzer - I have no idea what it would do but I presume there are some gotchas fr implementers	"Hypothetical. Consider how to do qlog big data, how to analyse large sets of logs and pick out anomalies"	Important (we use outside tooling and/or adapt open-source projects to our needs)	Often	Sometimes	Sometimes	Often	"More towards ""disable it for actual deployments and only use it for (local) debugging"" - logging creates pressure for certain production systems. Knowing what to log is hard to predict - enabling logging to fix specific issues is timebound and managable"	"yes, multiplexing and prioritization"	keep up the good work	Never	Of average importance		n/a	When the s$&% hit the fan		
3/24/2020 20:34:27	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), qlog (structured), custom format (structured) (e.g., in-house event tracing framework), custom format (unstructured) (e.g., command line output)"	The REDACTED implementation uses both structured and unstructured logging to maintain consistency with existing debugging capabilities. We also added qlog specifically for compatibility with other QUIC tools.		"We will maintain both. The unstructured logging is mostly for our use in developing the capability in REDACTED, with the structured logging intended for end users."	Very important	Absolutely essential	Absolutely essential	Not important	Very important	Of average importance	Of little importance	"Yes, but mainly structured logs (e.g., in combination with qvis)"		Very frequently	Very frequently	Occasionally	Occasionally	Occasionally	Never	Non-QUIC specific REDACTED performance profiling tools. Nothing that would be specifically relevant to QUIC	Occasionally			Very important (we have implemented a type of logging only to be able to use outside tooling)	Almost always	Almost always	Almost always	Almost always	Critical feature	Yes		Almost always	Very important	Frequently	Not considering removing it. Would only if there was an agreed upon standard replacement	So far it's been useful for compatibility testing and debugging		
3/24/2020 21:18:36	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), qlog (structured), custom format (unstructured) (e.g., command line output)"			"Currently, no.  At this company, logging has historically been low priority and our backlog is rather full.  Since we would have to design how our logging should look, and since we are able to debug issues effectively with our current logging, such a project would be low priority."	Of average importance	Not important	Very important	Of little importance	Absolutely essential	Of little importance	Absolutely essential	No or only rarely (I typically ask the other implementers to identify possible errors)		Very frequently	Occasionally	Rarely	Occasionally	Never	Never		Never	"A command-line Linux tool to explore qlog files.  Something like tcptrace, plus the ability to output a more concise list of packets sent/received (similar to tcpdump) would be awesome."		Very important (we have implemented a type of logging only to be able to use outside tooling)	Seldom	Often	Sometimes	Seldom	"Qlog is disabled by default but available for customers to use.  The intention is that customers could use qlog to provide traces for support requests as an alternative to providing logs or full packet traces.  AFAIK, there is no plan to promote this as a feature of our commercial product, just as an option for our support personnel to utilize."	"I feel it would be, though I can't think of any concrete examples :)"		Sometimes	Of average importance	Never	"I think we would consider removing qlog at some point in favor of using either a structured binary format or decrypted pcap files.  So far we haven't found much use with qlog, though we'll keep it around in case we find a good use case for it."			
3/24/2020 21:31:43	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), custom format (unstructured) (e.g., command line output)"		"I'm not sure whether qlog was available at the time i was starting my implementation. I did need some logging during development and developed a very simple lightweight logger. That satisfied my needs for a long time (actually, it still does). If have considered implementing qlog, but it will be quite some work and because my implementation does not even yet completely implement the (large) QUIC specification, it is not at top of my priority list. 
I do like the qlog format, and especially the tooling around the format, so i can imagine i will add qlog logging some day....."	"I'm not sure. The logger i'm currently using has categories that help me to get very specific logging for certain cases and i wouldn't like to loose that. Whether qlog could get me similar functionality would depend on tooling i guess. Note that to get the right logging, i now only have to set a command line parameter; as with WireShark, analysing log files is just a bit of a hassle (start the tool, select the right file, filter for the right information....), it's not necessarily complex but it is more work and every developer wants to minimise time spent in the run-test-learn cycle."	Of little importance	Of average importance	Very important	Very important	Of little importance	Not important	Very important	"Yes, but mainly structured logs (e.g., in combination with qvis)"		Very frequently	Frequently	Occasionally	Never	Never	Never		Never	Annotations in the log that explain certain behaviour. E.g. no ack sent because.... (ack delay? expecting pn-space to be discarded? only ack-ing every 2nd packet?). Or retransmitting data because.... (probe? packet lost?). Etc.		Very important: i would only consider implementing a structured logging format if tools are open source or otherwise free to use	Sometimes	Almost always	Never	Often	"I my implementation would implement qlog, i would certainly use that fact to promote the implementation."	"Not for me, because i don't develop other protocol implementations."		Never	Very important	Never	Probably not. Unless it would harm performance.	"Unstructured logs are usually useless for anyone except the implementors themselves, because it is usually hard to guess what certain log statements mean; actually, this is usually quite frustrating because you kind of understand it, but not exactly.
Mostly i check logs to find out if a packet my implementation sent, has been received and _try_ to find out why the server did not respond as i expected. Still with qlog this can be hard, because there is (was?) not so much info in the logs w.r.t. why a server did or did not take some action."		
3/24/2020 23:17:33	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), qlog (structured)"	"I am not sure *why* we decided on qlog format in REDACTED, but I am glad that we did. I believe that it offers us the ability to get a very granular look at the flow between endpoints in a way that is easier for application developers to understand. Being able to use the industry standard tools to visualize log files generated by these common formats is another great benefit for implementing logging in the qlog format. Obviously having the ability to also look at actual packet captures is helpful, but that will only help developers who have a certain setup and understanding on network systems. "	n/a	n/a	Not important	Very important	Of average importance	Absolutely essential	Not important	Very important	Very important	"I had no idea such shared log repositories existed! I absolutely will use them! That said, in terms of debugging, being able to communicate with authors of the libraries ""at the end other"" is invaluable, but that is something that can only meaningfully happen after understand the implementation of the local side and sharing these kinds of logs greatly improves that process."		Very frequently	Very frequently	Occasionally	Occasionally	Occasionally	Never	n/a	Never	"None that I can think of at this point, honestly. "	"I think that one of the problems I have faced in debugging interoperability issues is the lack of logging of ""exceptional"" conditions. The server(s) will gracefully handle bad packets, etc, and not emit a meaningful log. That makes it difficult for a novice developer like me to get a quick sense for where the problem is -- local implementation or peer implementation. "	Very important (we have implemented a type of logging only to be able to use outside tooling)	Almost always	Almost always	Almost always	Never	"I like structured logging because it provides the ability to share logs among participants and for the consortium of developers to build tools that work for everyone. I think text-based formats are expressive. Developers can use them without relying on special tools (like would be required if these logs were stored in binary format). At the same time, you *can* have those tools that add additional visualization and manipulation for the times where that is necessary. "	"Absolutely, 100%"	Thank you for synthesizing all this information! I can't wait to see the results. 	Almost always	Of average importance	Never	we did!	See above.		
3/25/2020 22:20:16	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), quictrace (structured), custom format (structured) (e.g., in-house event tracing framework), custom format (unstructured) (e.g., command line output)"	Better debugging		Unstructured logs are still useful when a human is reading through them	Of little importance	Very important	Very important	Of average importance	Of little importance	Very important	Absolutely essential	"Yes, but mainly unstructured logs"		Frequently	Rarely	Never	Never	Never	Rarely		Never			Important (we use outside tooling and/or adapt open-source projects to our needs)	Seldom	Seldom	Seldom	Seldom				Never	Absolutely essential	Occasionally				
3/28/2020 10:04:21	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), qlog (structured), custom format (unstructured) (e.g., command line output)"			"No, because structured logging like json requires much larger overhead than unstructured one."	Of average importance	Very important	Very important	Absolutely essential	Absolutely essential	Of little importance	Absolutely essential	"Yes, but mainly unstructured logs"		Occasionally	Rarely	Rarely	Rarely	Rarely	Never		Rarely			Important (we use outside tooling and/or adapt open-source projects to our needs)	Often	Often	Often	Often		"I think H2 can get benefit from structured logging, mainly investigating how priority mechanism affects scheduling DATA frames among multiple concurrent streams."		Often	Of average importance	Occasionally	I have no plan to remove it out.	When I cannot find any suspicious stuff in my implementation log.		
3/30/2020 19:49:53	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), qlog (structured), custom format (unstructured) (e.g., command line output)"		no	"There is implementation-specific stuff that also we'd like to see, not sure how structured logging would accommodate that."	Of little importance	Very important	Very important	Of little importance	Of little importance		Of average importance	No or only rarely (I typically ask the other implementers to identify possible errors)		Very frequently	Rarely	Occasionally	Never	Never	Never		Never			Very important (we have implemented a type of logging only to be able to use outside tooling)	Often	Often	Sometimes	Sometimes	"debugging use only, most likely"	quite possibly		Seldom	Very important	Never	no			
3/31/2020 3:08:25	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE)"		It was too much work. We plan to revisit qlog again.		Of little importance	Absolutely essential	Of average importance	Very important	Of average importance	Of average importance	Of average importance	No or only rarely (I typically ask the other implementers to identify possible errors)		Occasionally	Frequently	Rarely	Rarely	Never	Never		Rarely	A turn-key solution to enable logging and visualize the results.	Characterizing the pacing behavior in QUIC is hard without proper tools for logging and visualization.	Neutral (it does not influence our choice of logging format)	Often	Almost always	Often	Never	Quite important to help in our research.			Almost always	Of average importance	Rarely				
3/31/2020 3:46:48	"decrypted packet captures (exporting of TLS secrets via e.g., SSLKEYLOGFILE), qlog (structured), custom format (structured) (e.g., in-house event tracing framework), custom format (unstructured) (e.g., command line output)"	"Implemented a structured logging of congestion control parameters in a CSV file, one line per event, one column per parameters such as RTT, CWIN, etc. The goal was to performance analysis of congestion control, and tune algorithms."	"The main debugging tool is a text login of all incoming and outgoing packets, listing the frames in each packet, captured at client or server. 
Most of the debugging is done manually, by looking at the packet traces. This works very well for most issues, as it is generally sufficient to look at a few packets at the beginning or end of the trace, or maybe do a search for a specific frame type or event"	"I have implemented qlog as a two pass system: servers and clients can write compact binary logs, and then a tool can extract the qlog from the binary traces for relevant connections. The main reason was the big size of the qlog. The main drawback is that it does not provide ""real time logging"" that could be monitored on a parallel window.
I have almost reached content parity between the text log and the qlog, only missing the logging in qlog of error events. At that point, I might remove the old text log, but i will have to solve the ""real time availability"" issue. Qlog has its own limitations there, because the JSON is only syntactically correct after the whole connection has been logged.
"	Very important	Of little importance	Very important	Absolutely essential	Very important	Of average importance	Very important	"No or only rarely (I typically ask the other implementers to identify possible errors), Yes, but mainly structured logs (e.g., in combination with qvis)"		Very frequently	Occasionally	Frequently	Occasionally	Rarely	Never	"The same binary file that produces qlog can also produce CSV files tracing the congestion control parameters. I do use that very frequently. I also rely heavily on the simulation tools built in the picoquic test library to reproduce and understand specific issues. Typically, if a bug is found, I will devise and add a corresponding test in the test suite."	Rarely	Maybe the possibility to extract a scenario and replay it in a simulation.	"Not clear. When finding a problem, my first approach is to write tests"	Important (we use outside tooling and/or adapt open-source projects to our needs)	Often	Often	Almost always	Seldom	"It is nice for deployments, so users can understand what is happening. Better use a standard format than have to explain it to every network admin who uses picoquic..."	"Yes. For example, in DoQ, it would be interesting to have some logging of DNS level events."	"Your questionnaire does not mention the value of having standard logs, whether structured or not. Structure is nice because tools can be developed, but standards are very useful because they diminish the ""learning curve"" for users and network admins."	Seldom	Absolutely essential	Very frequently	If something better came along...			
3/31/2020 10:15:05	"custom format (unstructured) (e.g., command line output)"	Did not implement a logging format	I ended up using the default logging format provided by the Google quic implementation.	"Yes, because structured logging will presumably make analysis easier; without needing to clean the unstructured logs first."	Of average importance	Absolutely essential	Absolutely essential	Absolutely essential	Of little importance	Very important	Of average importance	"Yes, but mainly unstructured logs"		Very frequently	Rarely	Never	Never	Never	Never	N/A	Never	"Features to track cwnd, sequence numbers, rwnd and perhaps loss."	"We are interested in the way different quic implementations pace their packets. This is because we noticed unexplained bursts in previous video experiments we had carried. To efficiently investigate such a problem, it would be interesting to have a tool that accurately shows packet egress and ingress times, cwnd, rwnd and packet sequence numbers."	Important (we use outside tooling and/or adapt open-source projects to our needs)	Almost always	Often	Sometimes	Sometimes	N/A	"Given the size of DNS requests, it would be interesting to have a tool that compartmentalizes them and assign them flags that make it easier to detect anomalies, i.e. size of request being sent, or to identify blacklisted hosts (though I am not sure if tools with such capabilities already exist)"	N/A	Almost always	Absolutely essential	Never	N/A	With the logs provided by proto-quic (Google QUIC) we were able to observe packet transmission time and rwnd.		
3/31/2020 15:10:44	"custom format (unstructured) (e.g., command line output)"		"Our code is based on the chromium codebase and it would have been a pain to integrate logging into this huge beast of code - this is a lie in the sense that it would most certainly be possible to do so but we are a bit 'deadline driven' so it was always one of the two: every thing works - who needs logging? or: we need to fix this now! deadline is in two weeks, we don't have time to implement proper logging... we need to make use of the command line output that we have..."	I would love to but again with the 'deadline driven' work it is hard to sell spending weeks for implementing logging which is (for us) not providing material to publish and thus it (sadly) is always a second or third priority...	Of little importance	Absolutely essential	Very important	Very important	Of average importance	Absolutely essential	Of average importance	No or only rarely (I typically ask the other implementers to identify possible errors)		Very frequently	Occasionally	Never	Never	Never	Never	we have rudimentary tools (scripts) for plotting the logged data - they are primarily for plotting results for potential publications but were handy while debugging but mostly we either stared at console prints or used (do not laugh...) libre office calc to debug / see flows / behaviour 	Never	Basically: something like wireshark with proper decrypting of quic payloads - this should exist but so far we did not manage to easily get the crypto keys from our chrome system into wireshark...	We mostly had issues that were either payload related ( a proper tool to inspect packets is key there...) or related to the CC / ACK mechanism where a flow diagram would be really awesome (without tooling this resulted in us reading console prints of packet numbers and ack ranges - no fun...)	I wish I could say 'very important' but again: logging is sadly not a first class citizen but in case we du: open-source tools are the way to go!	Sometimes	Almost always	Seldom	Never	"If we have the chance to deploy any of our implementations where customers use it, we would certainly like to deploy structured logging as well - given the logging has no implications on performance (and there file size is very important)"	"YES! logging is way too important to miss out (I know this is a 'do as I say, not as I do'... but we very much acknowledge the need for proper logging"	"The problem is: for us it is hard as we right now are stuck with chrome's old quic implementation, so any time spend on implementing logging would be wasted, as they have already moved on to 'quiche'... from a 'giving back to the community' stand point it would make sense to spend time implementing something like qlog there but again back to the original issue: we never had a quiet enough time to spend efforts to put in logging into someone else's quic implementation (meaning, we are merely using it) if quiche provides support for qlog we would jump right in and extend the logging for our modifications but placing the logging in there in the first place is sadly not in the scope right now "	Almost always	Very important	Frequently	"Our logging currently is way too verbose to keep it in any kind of production version - if qlog or similar are small and fast enough to handle a production environment, I would definitely vote to keep it!"			
3/31/2020 21:36:35	"quictrace (structured), custom format (unstructured) (e.g., command line output)"	Available utilities.	We would like to use qlog.	Yes	Of average importance	Of average importance	Of little importance	Very important	Very important	Of little importance	Of little importance	No or only rarely (I typically ask the other implementers to identify possible errors)		Very frequently	Occasionally	Never	Never	Never	Rarely		Never		quictrace helps us to fight with performance problems.	Important (we use outside tooling and/or adapt open-source projects to our needs)	Never	Often	Often	Sometimes	We plan to use it mainly for debugging purposes and automated log processing.			Seldom	Of average importance	Very frequently	No.			
4/2/2020 22:02:18	"quictrace (structured), custom format (structured) (e.g., in-house event tracing framework)"	Large scale analysis across connections and size.			Very important	Very important	Of little importance	Of little importance	Very important	Not important	Very important	"Yes, both unstructured and structured logs"		Rarely	Rarely	Rarely	Never	Occasionally	Occasionally	"We have something similar to the quictrace time-sequence viewer, but it's integrated with internal infrastructure to find traces of interest."	Rarely	"Better tools for understanding congestion control and transport state(app-limited, time-sequence, CWND, flow control limited, etc) all on one interface."	qvis helped me locate a few cases I hadn't noticed where our QUIC implementation was sending FIN-only STREAM frames in a packet by themself.  But I have tons of examples of bugs identified by tooling.	Important (we use outside tooling and/or adapt open-source projects to our needs)	Seldom	Almost always	Often	Often	It's impossible to launch a new congestion controller(ie: BBRv1 or BBRv2) without it.	"Yes, I'd really like the tools I have for QUIC/H3 in HTTP 1.1 and HTTP/2 debugging."		Never	Absolutely essential	Frequently	Not unless it was to move to a different structured logging format.	"Debugging correctness issues, particularly unexpected connection closes or crashes."		
4/16/2020 20:35:06	"custom format (unstructured) (e.g., command line output)"	"Personally, I prefer structured formats, just haven't had time to do it yet"	"Speaking for myself, yes looking to add a structured format in the future."	"I think they achieve different goals, but we have an established logging infrastructure with support across the OS and I don't think we could eliminate that (and wouldn't benefit either way from doing so). The structured logging I expect to be not always-on and to be only used for event-tracing style of active capture, rather than built into the system infrastructure used by all components."	Very important	Of average importance	Very important	Of little importance	Of little importance	Very important	Of average importance	"Yes, but mainly unstructured logs"		Frequently	Very frequently	Rarely	Rarely	Rarely	Rarely		Rarely	"More HTTP/3 level visualizations, recovery/CC focused ones are great -- keep making those better too"	"Likely will, once we get there"	Very important (we have implemented a type of logging only to be able to use outside tooling)	Often	Almost always	Almost always	Seldom	Nothing to share about future plans	"Sure, if it can help the ecosystem of tooling do something better than tcptrace"		Sometimes	Very important	Occasionally	"If we had something better, sure, but otherwise I expect it will stay in and be disabled by default"	For debugging interop problems. Perf problems have been a bit harder to do with this