1 // Package runner implements a go/analysis runner. It makes heavy use
2 // of on-disk caching to reduce overall memory usage and to speed up
3 // repeat runs.
4 //
5 // # Public API
6 //
7 // A Runner maps a list of analyzers and package patterns to a list of
8 // results. Results provide access to diagnostics, directives, errors
9 // encountered, and information about packages. Results explicitly do
10 // not contain ASTs or type information. All position information is
11 // returned in the form of token.Position, not token.Pos. All work
12 // that requires access to the loaded representation of a package has
13 // to occur inside analyzers.
14 //
15 // # Planning and execution
16 //
17 // Analyzing packages is split into two phases: planning and
18 // execution.
19 //
20 // During planning, a directed acyclic graph of package dependencies
21 // is computed. We materialize the full graph so that we can execute
22 // the graph from the bottom up, without keeping unnecessary data in
23 // memory during a DFS and with simplified parallel execution.
24 //
25 // During execution, leaf nodes (nodes with no outstanding
26 // dependencies) get executed in parallel, bounded by a semaphore
27 // sized according to the number of CPUs. Conceptually, this happens
28 // in a loop, processing new leaf nodes as they appear, until no more
29 // nodes are left. In the actual implementation, nodes know their
30 // dependents, and the last dependency of a node to be processed is
31 // responsible for scheduling its dependent.
32 //
33 // The graph is rooted at a synthetic root node. Upon execution of the
34 // root node, the algorithm terminates.
35 //
36 // Analyzing a package repeats the same planning + execution steps,
37 // but this time on a graph of analyzers for the package. Parallel
38 // execution of individual analyzers is bounded by the same semaphore
39 // as executing packages.
40 //
41 // # Parallelism
42 //
43 // Actions are executed in parallel where the dependency graph allows.
44 // Overall parallelism is bounded by a semaphore, sized according to
45 // GOMAXPROCS. Each concurrently processed package takes up a
46 // token, as does each analyzer – but a package can always execute at
47 // least one analyzer, using the package's token.
48 //
49 // Depending on the overall shape of the graph, there may be GOMAXPROCS
50 // packages running a single analyzer each, a single package running
51 // GOMAXPROCS analyzers, or anything in between.
52 //
53 // Total memory consumption grows roughly linearly with the number of
54 // CPUs, while total execution time is inversely proportional to the
55 // number of CPUs. Overall, parallelism is affected by the shape of
56 // the dependency graph. A lot of inter-connected packages will see
57 // less parallelism than a lot of independent packages.
58 //
59 // # Caching
60 //
61 // The runner caches facts, directives and diagnostics in a
62 // content-addressable cache that is designed after Go's own cache.
63 // Additionally, it makes use of Go's export data.
64 //
65 // This cache not only speeds up repeat runs, it also reduces peak
66 // memory usage. When we've analyzed a package, we cache the results
67 // and drop them from memory. When a dependent needs any of this
68 // information, or when analysis is complete and we wish to render the
69 // results, the data gets loaded from disk again.
70 //
71 // Data only exists in memory when it is immediately needed, not
72 // retained for possible future uses. This trades increased CPU usage
73 // for reduced memory usage. A single dependency may be loaded many
74 // times over, but it greatly reduces peak memory usage, as an
75 // arbitrary amount of time may pass between analyzing a dependency
76 // and its dependent, during which other packages will be processed.
77 package runner
78 79 // OPT(dh): we could reduce disk storage usage of cached data by
80 // compressing it, either directly at the cache layer, or by feeding
81 // compressed data to the cache. Of course doing so may negatively
82 // affect CPU usage, and there are lower hanging fruit, such as
83 // needing to cache less data in the first place.
84 85 // OPT(dh): right now, each package is analyzed completely
86 // independently. Each package loads all of its dependencies from
87 // export data and cached facts. If we have two packages A and B,
88 // which both depend on C, and which both get analyzed in parallel,
89 // then C will be loaded twice. This wastes CPU time and memory. It
90 // would be nice if we could reuse a single C for the analysis of both
91 // A and B.
92 //
93 // We can't reuse the actual types.Package or facts, because each
94 // package gets its own token.FileSet. Sharing a global FileSet has
95 // several drawbacks, including increased memory usage and running the
96 // risk of running out of FileSet address space.
97 //
98 // We could however avoid loading the same raw export data from disk
99 // twice, as well as deserializing gob data twice. One possible
100 // solution would be a duplicate-suppressing in-memory cache that
101 // caches data for a limited amount of time. When the same package
102 // needs to be loaded twice in close succession, we can reuse work,
103 // without holding unnecessary data in memory for an extended period
104 // of time.
105 //
106 // We would likely need to do extensive benchmarking to figure out how
107 // long to keep data around to find a sweet spot where we reduce CPU
108 // load without increasing memory usage.
109 //
110 // We can probably populate the cache after we've analyzed a package,
111 // on the assumption that it will have to be loaded again in the near
112 // future.
113 114 import (
115 "bytes"
116 "encoding/gob"
117 "fmt"
118 "go/token"
119 "go/types"
120 "io"
121 "os"
122 "reflect"
123 "runtime"
124 "sort"
125 "strings"
126 "sync/atomic"
127 "time"
128 129 "honnef.co/go/tools/analysis/lint"
130 "honnef.co/go/tools/analysis/report"
131 "honnef.co/go/tools/config"
132 "honnef.co/go/tools/go/loader"
133 tsync "honnef.co/go/tools/internal/sync"
134 "honnef.co/go/tools/lintcmd/cache"
135 "honnef.co/go/tools/unused"
136 137 "golang.org/x/tools/go/analysis"
138 "golang.org/x/tools/go/packages"
139 "golang.org/x/tools/go/types/objectpath"
140 )
141 142 const sanityCheck = false
143 144 // Diagnostic is like go/analysis.Diagnostic, but with all token.Pos resolved to token.Position.
145 type Diagnostic struct {
146 Position token.Position
147 End token.Position
148 Category string
149 Message string
150 151 SuggestedFixes []SuggestedFix
152 Related []RelatedInformation
153 }
154 155 // RelatedInformation provides additional context for a diagnostic.
156 type RelatedInformation struct {
157 Position token.Position
158 End token.Position
159 Message string
160 }
161 162 type SuggestedFix struct {
163 Message string
164 TextEdits []TextEdit
165 }
166 167 type TextEdit struct {
168 Position token.Position
169 End token.Position
170 NewText []byte
171 }
172 173 // A Result describes the result of analyzing a single package.
174 //
175 // It holds references to cached diagnostics and directives. They can
176 // be loaded on demand with the Load method.
177 type Result struct {
178 Package *loader.PackageSpec
179 Config config.Config
180 Initial bool
181 Skipped bool
182 183 Failed bool
184 Errors []error
185 // Action results, path to file
186 results string
187 // Results relevant to testing, only set when test mode is enabled, path to file
188 testData string
189 }
190 191 type SerializedDirective struct {
192 Command string
193 Arguments []string
194 // The position of the comment
195 DirectivePosition token.Position
196 // The position of the node that the comment is attached to
197 NodePosition token.Position
198 }
199 200 func serializeDirective(dir lint.Directive, fset *token.FileSet) SerializedDirective {
201 return SerializedDirective{
202 Command: dir.Command,
203 Arguments: dir.Arguments,
204 DirectivePosition: report.DisplayPosition(fset, dir.Directive.Pos()),
205 NodePosition: report.DisplayPosition(fset, dir.Node.Pos()),
206 }
207 }
208 209 type ResultData struct {
210 Directives []SerializedDirective
211 Diagnostics []Diagnostic
212 Unused unused.Result
213 }
214 215 func (r Result) Load() (ResultData, error) {
216 if r.Failed {
217 panic("Load called on failed Result")
218 }
219 if r.results == "" {
220 // this package was only a dependency
221 return ResultData{}, nil
222 }
223 f, err := os.Open(r.results)
224 if err != nil {
225 return ResultData{}, fmt.Errorf("failed loading result: %w", err)
226 }
227 defer f.Close()
228 var out ResultData
229 err = gob.NewDecoder(f).Decode(&out)
230 return out, err
231 }
232 233 // TestData contains extra information about analysis runs that is only available in test mode.
234 type TestData struct {
235 // Facts contains facts produced by analyzers for a package.
236 // Unlike vetx, this list only contains facts specific to this package,
237 // not all facts for the transitive closure of dependencies.
238 Facts []TestFact
239 // List of files that were part of the package.
240 Files []string
241 }
242 243 // LoadTest returns data relevant to testing.
244 // It should only be called if Runner.TestMode was set to true.
245 func (r Result) LoadTest() (TestData, error) {
246 if r.Failed {
247 panic("Load called on failed Result")
248 }
249 if r.results == "" {
250 // this package was only a dependency
251 return TestData{}, nil
252 }
253 f, err := os.Open(r.testData)
254 if err != nil {
255 return TestData{}, fmt.Errorf("failed loading test data: %w", err)
256 }
257 defer f.Close()
258 var out TestData
259 err = gob.NewDecoder(f).Decode(&out)
260 return out, err
261 }
262 263 type action interface {
264 Deps() []action
265 Triggers() []action
266 DecrementPending() bool
267 MarkFailed()
268 IsFailed() bool
269 AddError(error)
270 }
271 272 type baseAction struct {
273 // Action description
274 275 deps []action
276 triggers []action
277 pending uint32
278 279 // Action results
280 281 // failed is set to true if the action couldn't be processed. This
282 // may either be due to an error specific to this action, in
283 // which case the errors field will be populated, or due to a
284 // dependency being marked as failed, in which case errors will be
285 // empty.
286 failed bool
287 errors []error
288 }
289 290 func (act *baseAction) Deps() []action { return act.deps }
291 func (act *baseAction) Triggers() []action { return act.triggers }
292 func (act *baseAction) DecrementPending() bool {
293 return atomic.AddUint32(&act.pending, ^uint32(0)) == 0
294 }
295 func (act *baseAction) MarkFailed() { act.failed = true }
296 func (act *baseAction) IsFailed() bool { return act.failed }
297 func (act *baseAction) AddError(err error) { act.errors = append(act.errors, err) }
298 299 // packageAction describes the act of loading a package, fully
300 // analyzing it, and storing the results.
301 type packageAction struct {
302 baseAction
303 304 // Action description
305 Package *loader.PackageSpec
306 factsOnly bool
307 hash cache.ActionID
308 309 // Action results
310 cfg config.Config
311 vetx string
312 results string
313 testData string
314 skipped bool
315 }
316 317 func (act *packageAction) String() string {
318 return fmt.Sprintf("packageAction(%s)", act.Package)
319 }
320 321 type objectFact struct {
322 fact analysis.Fact
323 // TODO(dh): why do we store the objectpath when producing the
324 // fact? Is it just for the sanity checking, which compares the
325 // stored path with a path recomputed from objectFactKey.Obj?
326 path objectpath.Path
327 }
328 329 type objectFactKey struct {
330 Obj types.Object
331 Type reflect.Type
332 }
333 334 type packageFactKey struct {
335 Pkg *types.Package
336 Type reflect.Type
337 }
338 339 type gobFact struct {
340 PkgPath string
341 ObjPath string
342 Fact analysis.Fact
343 }
344 345 // TestFact is a serialization of facts that is specific to the test mode.
346 type TestFact struct {
347 ObjectName string
348 Position token.Position
349 FactString string
350 Analyzer string
351 }
352 353 // analyzerAction describes the act of analyzing a package with a
354 // single analyzer.
355 type analyzerAction struct {
356 baseAction
357 358 // Action description
359 360 Analyzer *analysis.Analyzer
361 362 // Action results
363 364 // We can store actual results here without worrying about memory
365 // consumption because analyzer actions get garbage collected once
366 // a package has been fully analyzed.
367 Result interface{}
368 Diagnostics []Diagnostic
369 ObjectFacts map[objectFactKey]objectFact
370 PackageFacts map[packageFactKey]analysis.Fact
371 Pass *analysis.Pass
372 }
373 374 func (act *analyzerAction) String() string {
375 return fmt.Sprintf("analyzerAction(%s)", act.Analyzer)
376 }
377 378 // A Runner executes analyzers on packages.
379 type Runner struct {
380 Stats Stats
381 GoVersion string
382 383 // If set to true, Runner will populate results with data relevant to testing analyzers
384 TestMode bool
385 386 // Config that gets merged with per-package configs
387 cfg config.Config
388 cache *cache.Cache
389 semaphore tsync.Semaphore
390 }
391 392 type subrunner struct {
393 *Runner
394 analyzers []*analysis.Analyzer
395 factAnalyzers []*analysis.Analyzer
396 analyzerNames string
397 cache *cache.Cache
398 }
399 400 // New returns a new Runner.
401 func New(cfg config.Config, c *cache.Cache) (*Runner, error) {
402 return &Runner{
403 cfg: cfg,
404 cache: c,
405 semaphore: tsync.NewSemaphore(runtime.GOMAXPROCS(0)),
406 }, nil
407 }
408 409 func newSubrunner(r *Runner, analyzers []*analysis.Analyzer) *subrunner {
410 analyzerNames := make([]string, len(analyzers))
411 for i, a := range analyzers {
412 analyzerNames[i] = a.Name
413 }
414 sort.Strings(analyzerNames)
415 416 var factAnalyzers []*analysis.Analyzer
417 for _, a := range analyzers {
418 if len(a.FactTypes) > 0 {
419 factAnalyzers = append(factAnalyzers, a)
420 }
421 }
422 return &subrunner{
423 Runner: r,
424 analyzers: analyzers,
425 factAnalyzers: factAnalyzers,
426 analyzerNames: strings.Join(analyzerNames, ","),
427 cache: r.cache,
428 }
429 }
430 431 func newPackageActionRoot(pkg *loader.PackageSpec, cache map[*loader.PackageSpec]*packageAction) *packageAction {
432 a := newPackageAction(pkg, cache)
433 a.factsOnly = false
434 return a
435 }
436 437 func newPackageAction(pkg *loader.PackageSpec, cache map[*loader.PackageSpec]*packageAction) *packageAction {
438 if a, ok := cache[pkg]; ok {
439 return a
440 }
441 442 a := &packageAction{
443 Package: pkg,
444 factsOnly: true, // will be overwritten by any call to Action
445 }
446 cache[pkg] = a
447 448 if len(pkg.Errors) > 0 {
449 a.errors = make([]error, len(pkg.Errors))
450 for i, err := range pkg.Errors {
451 a.errors[i] = err
452 }
453 a.failed = true
454 455 // We don't need to process our imports if this package is
456 // already broken.
457 return a
458 }
459 460 a.deps = make([]action, 0, len(pkg.Imports))
461 for _, dep := range pkg.Imports {
462 depa := newPackageAction(dep, cache)
463 depa.triggers = append(depa.triggers, a)
464 a.deps = append(a.deps, depa)
465 466 if depa.failed {
467 a.failed = true
468 }
469 }
470 // sort dependencies because the list of dependencies is part of
471 // the cache key
472 sort.Slice(a.deps, func(i, j int) bool {
473 return a.deps[i].(*packageAction).Package.ID < a.deps[j].(*packageAction).Package.ID
474 })
475 476 a.pending = uint32(len(a.deps))
477 478 return a
479 }
480 481 func newAnalyzerAction(an *analysis.Analyzer, cache map[*analysis.Analyzer]*analyzerAction) *analyzerAction {
482 if a, ok := cache[an]; ok {
483 return a
484 }
485 486 a := &analyzerAction{
487 Analyzer: an,
488 ObjectFacts: map[objectFactKey]objectFact{},
489 PackageFacts: map[packageFactKey]analysis.Fact{},
490 }
491 cache[an] = a
492 for _, dep := range an.Requires {
493 depa := newAnalyzerAction(dep, cache)
494 depa.triggers = append(depa.triggers, a)
495 a.deps = append(a.deps, depa)
496 }
497 a.pending = uint32(len(a.deps))
498 return a
499 }
500 501 func getCachedFiles(cache *cache.Cache, ids []cache.ActionID, out []*string) error {
502 for i, id := range ids {
503 var err error
504 *out[i], _, err = cache.GetFile(id)
505 if err != nil {
506 return err
507 }
508 }
509 return nil
510 }
511 512 func (r *subrunner) do(act action) error {
513 a := act.(*packageAction)
514 defer func() {
515 r.Stats.finishPackage()
516 if !a.factsOnly {
517 r.Stats.finishInitialPackage()
518 }
519 }()
520 521 // compute hash of action
522 a.cfg = a.Package.Config.Merge(r.cfg)
523 h := r.cache.NewHash("staticcheck " + a.Package.PkgPath)
524 525 // Note that we do not filter the list of analyzers by the
526 // package's configuration. We don't allow configuration to
527 // accidentally break dependencies between analyzers, and it's
528 // easier to always run all checks and filter the output. This
529 // also makes cached data more reusable.
530 531 // OPT(dh): not all changes in configuration invalidate cached
532 // data. specifically, when a.factsOnly == true, we only care
533 // about checks that produce facts, and settings that affect those
534 // checks.
535 536 // Config used for constructing the hash; this config doesn't have
537 // Checks populated, because we always run all checks.
538 //
539 // This even works for users who add custom checks, because we include the binary's hash.
540 hashCfg := a.cfg
541 hashCfg.Checks = nil
542 // note that we don't hash staticcheck's version; it is set as the
543 // salt by a package main.
544 fmt.Fprintf(h, "cfg %#v\n", hashCfg)
545 fmt.Fprintf(h, "pkg %x\n", a.Package.Hash)
546 fmt.Fprintf(h, "analyzers %s\n", r.analyzerNames)
547 fmt.Fprintf(h, "go %s\n", r.GoVersion)
548 fmt.Fprintf(h, "env godebug %q\n", os.Getenv("GODEBUG"))
549 550 // OPT(dh): do we actually need to hash vetx? can we not assume
551 // that for identical inputs, staticcheck will produce identical
552 // vetx?
553 for _, dep := range a.deps {
554 dep := dep.(*packageAction)
555 vetxHash, err := cache.FileHash(dep.vetx)
556 if err != nil {
557 return fmt.Errorf("failed computing hash: %w", err)
558 }
559 fmt.Fprintf(h, "vetout %q %x\n", dep.Package.PkgPath, vetxHash)
560 }
561 a.hash = cache.ActionID(h.Sum())
562 563 // try to fetch hashed data
564 ids := make([]cache.ActionID, 0, 2)
565 ids = append(ids, cache.Subkey(a.hash, "vetx"))
566 if !a.factsOnly {
567 ids = append(ids, cache.Subkey(a.hash, "results"))
568 if r.TestMode {
569 ids = append(ids, cache.Subkey(a.hash, "testdata"))
570 }
571 }
572 if err := getCachedFiles(r.cache, ids, []*string{&a.vetx, &a.results, &a.testData}); err != nil {
573 result, err := r.doUncached(a)
574 if err != nil {
575 return err
576 }
577 if a.failed {
578 return nil
579 }
580 581 a.skipped = result.skipped
582 583 // OPT(dh) instead of collecting all object facts and encoding
584 // them after analysis finishes, we could encode them as we
585 // go. however, that would require some locking.
586 //
587 // OPT(dh): We could sort gobFacts for more consistent output,
588 // but it doesn't matter. The hash of a package includes all
589 // of its files, so whether the vetx hash changes or not, a
590 // change to a package requires re-analyzing all dependents,
591 // even if the vetx data stayed the same. See also the note at
592 // the top of loader/hash.go.
593 594 tf := &bytes.Buffer{}
595 enc := gob.NewEncoder(tf)
596 for _, gf := range result.facts {
597 if err := enc.Encode(gf); err != nil {
598 return fmt.Errorf("failed gob encoding data: %w", err)
599 }
600 }
601 602 a.vetx, err = r.writeCacheReader(a, "vetx", bytes.NewReader(tf.Bytes()))
603 if err != nil {
604 return err
605 }
606 607 if a.factsOnly {
608 return nil
609 }
610 611 var out ResultData
612 out.Directives = make([]SerializedDirective, len(result.dirs))
613 for i, dir := range result.dirs {
614 out.Directives[i] = serializeDirective(dir, result.lpkg.Fset)
615 }
616 617 out.Diagnostics = result.diags
618 out.Unused = result.unused
619 a.results, err = r.writeCacheGob(a, "results", out)
620 if err != nil {
621 return err
622 }
623 624 if r.TestMode {
625 out := TestData{
626 Facts: result.testFacts,
627 Files: result.lpkg.GoFiles,
628 }
629 a.testData, err = r.writeCacheGob(a, "testdata", out)
630 if err != nil {
631 return err
632 }
633 }
634 }
635 return nil
636 }
637 638 // ActiveWorkers returns the number of currently running workers.
639 func (r *Runner) ActiveWorkers() int {
640 return r.semaphore.Len()
641 }
642 643 // TotalWorkers returns the maximum number of possible workers.
644 func (r *Runner) TotalWorkers() int {
645 return r.semaphore.Cap()
646 }
647 648 func (r *Runner) writeCacheReader(a *packageAction, kind string, rs io.ReadSeeker) (string, error) {
649 h := cache.Subkey(a.hash, kind)
650 out, _, err := r.cache.Put(h, rs)
651 if err != nil {
652 return "", fmt.Errorf("failed caching data: %w", err)
653 }
654 return r.cache.OutputFile(out), nil
655 }
656 657 func (r *Runner) writeCacheGob(a *packageAction, kind string, data interface{}) (string, error) {
658 f, err := os.CreateTemp("", "staticcheck")
659 if err != nil {
660 return "", err
661 }
662 defer f.Close()
663 os.Remove(f.Name())
664 if err := gob.NewEncoder(f).Encode(data); err != nil {
665 return "", fmt.Errorf("failed gob encoding data: %w", err)
666 }
667 if _, err := f.Seek(0, io.SeekStart); err != nil {
668 return "", err
669 }
670 return r.writeCacheReader(a, kind, f)
671 }
672 673 type packageActionResult struct {
674 facts []gobFact
675 diags []Diagnostic
676 unused unused.Result
677 dirs []lint.Directive
678 lpkg *loader.Package
679 skipped bool
680 681 // Only set when using test mode
682 testFacts []TestFact
683 }
684 685 func (r *subrunner) doUncached(a *packageAction) (packageActionResult, error) {
686 // OPT(dh): for a -> b; c -> b; if both a and b are being
687 // processed concurrently, we shouldn't load b's export data
688 // twice.
689 690 pkg, _, err := loader.Load(a.Package, &loader.Options{GoVersion: r.GoVersion})
691 if err != nil {
692 return packageActionResult{}, err
693 }
694 695 if len(pkg.Errors) > 0 {
696 // this handles errors that occurred during type-checking the
697 // package in loader.Load
698 for _, err := range pkg.Errors {
699 a.errors = append(a.errors, err)
700 }
701 a.failed = true
702 return packageActionResult{}, nil
703 }
704 705 if len(pkg.Syntax) == 0 && pkg.PkgPath != "unsafe" {
706 return packageActionResult{lpkg: pkg, skipped: true}, nil
707 }
708 709 // OPT(dh): instead of parsing directives twice (twice because
710 // U1000 depends on the facts.Directives analyzer), reuse the
711 // existing result
712 var dirs []lint.Directive
713 if !a.factsOnly {
714 dirs = lint.ParseDirectives(pkg.Syntax, pkg.Fset)
715 }
716 res, err := r.runAnalyzers(a, pkg)
717 718 return packageActionResult{
719 facts: res.facts,
720 testFacts: res.testFacts,
721 diags: res.diagnostics,
722 unused: res.unused,
723 dirs: dirs,
724 lpkg: pkg,
725 }, err
726 }
727 728 func pkgPaths(root *types.Package) map[string]*types.Package {
729 out := map[string]*types.Package{}
730 var dfs func(*types.Package)
731 dfs = func(pkg *types.Package) {
732 if _, ok := out[pkg.Path()]; ok {
733 return
734 }
735 out[pkg.Path()] = pkg
736 for _, imp := range pkg.Imports() {
737 dfs(imp)
738 }
739 }
740 dfs(root)
741 return out
742 }
743 744 func (r *Runner) loadFacts(root *types.Package, dep *packageAction, objFacts map[objectFactKey]objectFact, pkgFacts map[packageFactKey]analysis.Fact) error {
745 // Load facts of all imported packages
746 vetx, err := os.Open(dep.vetx)
747 if err != nil {
748 return fmt.Errorf("failed loading cached facts: %w", err)
749 }
750 defer vetx.Close()
751 752 pathToPkg := pkgPaths(root)
753 dec := gob.NewDecoder(vetx)
754 for {
755 var gf gobFact
756 err := dec.Decode(&gf)
757 if err != nil {
758 if err == io.EOF {
759 break
760 }
761 return fmt.Errorf("failed loading cached facts: %w", err)
762 }
763 764 pkg, ok := pathToPkg[gf.PkgPath]
765 if !ok {
766 continue
767 }
768 if gf.ObjPath == "" {
769 pkgFacts[packageFactKey{
770 Pkg: pkg,
771 Type: reflect.TypeOf(gf.Fact),
772 }] = gf.Fact
773 } else {
774 obj, err := objectpath.Object(pkg, objectpath.Path(gf.ObjPath))
775 if err != nil {
776 continue
777 }
778 objFacts[objectFactKey{
779 Obj: obj,
780 Type: reflect.TypeOf(gf.Fact),
781 }] = objectFact{gf.Fact, objectpath.Path(gf.ObjPath)}
782 }
783 }
784 return nil
785 }
786 787 func genericHandle(a action, root action, queue chan action, sem *tsync.Semaphore, exec func(a action) error) {
788 if a == root {
789 close(queue)
790 if sem != nil {
791 sem.Release()
792 }
793 return
794 }
795 if !a.IsFailed() {
796 // the action may have already been marked as failed during
797 // construction of the action graph, for example because of
798 // unresolved imports.
799 800 for _, dep := range a.Deps() {
801 if dep.IsFailed() {
802 // One of our dependencies failed, so mark this package as
803 // failed and bail. We don't need to record an error for
804 // this package, the relevant error will have been
805 // reported by the first package in the chain that failed.
806 a.MarkFailed()
807 break
808 }
809 }
810 }
811 812 if !a.IsFailed() {
813 if err := exec(a); err != nil {
814 a.MarkFailed()
815 a.AddError(err)
816 }
817 }
818 if sem != nil {
819 sem.Release()
820 }
821 822 for _, t := range a.Triggers() {
823 if t.DecrementPending() {
824 queue <- t
825 }
826 }
827 }
828 829 type analyzerRunner struct {
830 pkg *loader.Package
831 // object facts of our dependencies; may contain facts of
832 // analyzers other than the current one
833 depObjFacts map[objectFactKey]objectFact
834 // package facts of our dependencies; may contain facts of
835 // analyzers other than the current one
836 depPkgFacts map[packageFactKey]analysis.Fact
837 factsOnly bool
838 839 stats *Stats
840 }
841 842 func (ar *analyzerRunner) do(act action) error {
843 a := act.(*analyzerAction)
844 results := map[*analysis.Analyzer]interface{}{}
845 // TODO(dh): does this have to be recursive?
846 for _, dep := range a.deps {
847 dep := dep.(*analyzerAction)
848 results[dep.Analyzer] = dep.Result
849 }
850 // OPT(dh): cache factTypes, it is the same for all packages for a given analyzer
851 //
852 // OPT(dh): do we need the factTypes map? most analyzers have 0-1
853 // fact types. iterating over the slice is probably faster than
854 // indexing a map.
855 factTypes := map[reflect.Type]struct{}{}
856 for _, typ := range a.Analyzer.FactTypes {
857 factTypes[reflect.TypeOf(typ)] = struct{}{}
858 }
859 filterFactType := func(typ reflect.Type) bool {
860 _, ok := factTypes[typ]
861 return ok
862 }
863 a.Pass = &analysis.Pass{
864 Analyzer: a.Analyzer,
865 Fset: ar.pkg.Fset,
866 Files: ar.pkg.Syntax,
867 OtherFiles: ar.pkg.OtherFiles,
868 Pkg: ar.pkg.Types,
869 TypesInfo: ar.pkg.TypesInfo,
870 TypesSizes: ar.pkg.TypesSizes,
871 Report: func(diag analysis.Diagnostic) {
872 if !ar.factsOnly {
873 if diag.Category == "" {
874 diag.Category = a.Analyzer.Name
875 }
876 d := Diagnostic{
877 Position: report.DisplayPosition(ar.pkg.Fset, diag.Pos),
878 End: report.DisplayPosition(ar.pkg.Fset, diag.End),
879 Category: diag.Category,
880 Message: diag.Message,
881 }
882 for _, sugg := range diag.SuggestedFixes {
883 s := SuggestedFix{
884 Message: sugg.Message,
885 }
886 for _, edit := range sugg.TextEdits {
887 s.TextEdits = append(s.TextEdits, TextEdit{
888 Position: report.DisplayPosition(ar.pkg.Fset, edit.Pos),
889 End: report.DisplayPosition(ar.pkg.Fset, edit.End),
890 NewText: edit.NewText,
891 })
892 }
893 d.SuggestedFixes = append(d.SuggestedFixes, s)
894 }
895 for _, rel := range diag.Related {
896 d.Related = append(d.Related, RelatedInformation{
897 Position: report.DisplayPosition(ar.pkg.Fset, rel.Pos),
898 End: report.DisplayPosition(ar.pkg.Fset, rel.End),
899 Message: rel.Message,
900 })
901 }
902 a.Diagnostics = append(a.Diagnostics, d)
903 }
904 },
905 ResultOf: results,
906 ImportObjectFact: func(obj types.Object, fact analysis.Fact) bool {
907 key := objectFactKey{
908 Obj: obj,
909 Type: reflect.TypeOf(fact),
910 }
911 if f, ok := ar.depObjFacts[key]; ok {
912 reflect.ValueOf(fact).Elem().Set(reflect.ValueOf(f.fact).Elem())
913 return true
914 } else if f, ok := a.ObjectFacts[key]; ok {
915 reflect.ValueOf(fact).Elem().Set(reflect.ValueOf(f.fact).Elem())
916 return true
917 }
918 return false
919 },
920 ImportPackageFact: func(pkg *types.Package, fact analysis.Fact) bool {
921 key := packageFactKey{
922 Pkg: pkg,
923 Type: reflect.TypeOf(fact),
924 }
925 if f, ok := ar.depPkgFacts[key]; ok {
926 reflect.ValueOf(fact).Elem().Set(reflect.ValueOf(f).Elem())
927 return true
928 } else if f, ok := a.PackageFacts[key]; ok {
929 reflect.ValueOf(fact).Elem().Set(reflect.ValueOf(f).Elem())
930 return true
931 }
932 return false
933 },
934 ExportObjectFact: func(obj types.Object, fact analysis.Fact) {
935 key := objectFactKey{
936 Obj: obj,
937 Type: reflect.TypeOf(fact),
938 }
939 path, _ := objectpath.For(obj)
940 a.ObjectFacts[key] = objectFact{fact, path}
941 },
942 ExportPackageFact: func(fact analysis.Fact) {
943 key := packageFactKey{
944 Pkg: ar.pkg.Types,
945 Type: reflect.TypeOf(fact),
946 }
947 a.PackageFacts[key] = fact
948 },
949 AllPackageFacts: func() []analysis.PackageFact {
950 out := make([]analysis.PackageFact, 0, len(ar.depPkgFacts)+len(a.PackageFacts))
951 for key, fact := range ar.depPkgFacts {
952 out = append(out, analysis.PackageFact{
953 Package: key.Pkg,
954 Fact: fact,
955 })
956 }
957 for key, fact := range a.PackageFacts {
958 out = append(out, analysis.PackageFact{
959 Package: key.Pkg,
960 Fact: fact,
961 })
962 }
963 return out
964 },
965 AllObjectFacts: func() []analysis.ObjectFact {
966 out := make([]analysis.ObjectFact, 0, len(ar.depObjFacts)+len(a.ObjectFacts))
967 for key, fact := range ar.depObjFacts {
968 if filterFactType(key.Type) {
969 out = append(out, analysis.ObjectFact{
970 Object: key.Obj,
971 Fact: fact.fact,
972 })
973 }
974 }
975 for key, fact := range a.ObjectFacts {
976 if filterFactType(key.Type) {
977 out = append(out, analysis.ObjectFact{
978 Object: key.Obj,
979 Fact: fact.fact,
980 })
981 }
982 }
983 return out
984 },
985 }
986 987 t := time.Now()
988 res, err := a.Analyzer.Run(a.Pass)
989 ar.stats.measureAnalyzer(a.Analyzer, ar.pkg.PackageSpec, time.Since(t))
990 if err != nil {
991 return err
992 }
993 a.Result = res
994 return nil
995 }
996 997 type analysisResult struct {
998 facts []gobFact
999 diagnostics []Diagnostic
1000 unused unused.Result
1001 1002 // Only set when using test mode
1003 testFacts []TestFact
1004 }
1005 1006 func (r *subrunner) runAnalyzers(pkgAct *packageAction, pkg *loader.Package) (analysisResult, error) {
1007 depObjFacts := map[objectFactKey]objectFact{}
1008 depPkgFacts := map[packageFactKey]analysis.Fact{}
1009 1010 for _, dep := range pkgAct.deps {
1011 if err := r.loadFacts(pkg.Types, dep.(*packageAction), depObjFacts, depPkgFacts); err != nil {
1012 return analysisResult{}, err
1013 }
1014 }
1015 1016 root := &analyzerAction{}
1017 var analyzers []*analysis.Analyzer
1018 if pkgAct.factsOnly {
1019 // When analyzing non-initial packages, we only care about
1020 // analyzers that produce facts.
1021 analyzers = r.factAnalyzers
1022 } else {
1023 analyzers = r.analyzers
1024 }
1025 1026 all := map[*analysis.Analyzer]*analyzerAction{}
1027 for _, a := range analyzers {
1028 a := newAnalyzerAction(a, all)
1029 root.deps = append(root.deps, a)
1030 a.triggers = append(a.triggers, root)
1031 }
1032 root.pending = uint32(len(root.deps))
1033 1034 ar := &analyzerRunner{
1035 pkg: pkg,
1036 factsOnly: pkgAct.factsOnly,
1037 depObjFacts: depObjFacts,
1038 depPkgFacts: depPkgFacts,
1039 stats: &r.Stats,
1040 }
1041 queue := make(chan action, len(all))
1042 for _, a := range all {
1043 if len(a.Deps()) == 0 {
1044 queue <- a
1045 }
1046 }
1047 1048 // Don't hang if there are no analyzers to run; for example
1049 // because we are analyzing a dependency but have no analyzers
1050 // that produce facts.
1051 if len(all) == 0 {
1052 close(queue)
1053 }
1054 for item := range queue {
1055 b := r.semaphore.AcquireMaybe()
1056 if b {
1057 go genericHandle(item, root, queue, &r.semaphore, ar.do)
1058 } else {
1059 // the semaphore is exhausted; run the analysis under the
1060 // token we've acquired for analyzing the package.
1061 genericHandle(item, root, queue, nil, ar.do)
1062 }
1063 }
1064 1065 var unusedResult unused.Result
1066 for _, a := range all {
1067 if a != root && a.Analyzer.Name == "U1000" && !a.failed {
1068 // TODO(dh): figure out a clean abstraction, instead of
1069 // special-casing U1000.
1070 unusedResult = a.Result.(unused.Result)
1071 }
1072 1073 for key, fact := range a.ObjectFacts {
1074 depObjFacts[key] = fact
1075 }
1076 for key, fact := range a.PackageFacts {
1077 depPkgFacts[key] = fact
1078 }
1079 }
1080 1081 // OPT(dh): cull objects not reachable via the exported closure
1082 var testFacts []TestFact
1083 gobFacts := make([]gobFact, 0, len(depObjFacts)+len(depPkgFacts))
1084 for key, fact := range depObjFacts {
1085 if fact.path == "" {
1086 continue
1087 }
1088 if sanityCheck {
1089 p, _ := objectpath.For(key.Obj)
1090 if p != fact.path {
1091 panic(fmt.Sprintf("got different object paths for %v. old: %q new: %q", key.Obj, fact.path, p))
1092 }
1093 }
1094 gf := gobFact{
1095 PkgPath: key.Obj.Pkg().Path(),
1096 ObjPath: string(fact.path),
1097 Fact: fact.fact,
1098 }
1099 gobFacts = append(gobFacts, gf)
1100 }
1101 1102 for key, fact := range depPkgFacts {
1103 gf := gobFact{
1104 PkgPath: key.Pkg.Path(),
1105 Fact: fact,
1106 }
1107 gobFacts = append(gobFacts, gf)
1108 }
1109 1110 if r.TestMode {
1111 for _, a := range all {
1112 for key, fact := range a.ObjectFacts {
1113 tgf := TestFact{
1114 ObjectName: key.Obj.Name(),
1115 Position: pkg.Fset.Position(key.Obj.Pos()),
1116 FactString: fmt.Sprint(fact.fact),
1117 Analyzer: a.Analyzer.Name,
1118 }
1119 testFacts = append(testFacts, tgf)
1120 }
1121 1122 for _, fact := range a.PackageFacts {
1123 tgf := TestFact{
1124 ObjectName: "",
1125 Position: pkg.Fset.Position(pkg.Syntax[0].Pos()),
1126 FactString: fmt.Sprint(fact),
1127 Analyzer: a.Analyzer.Name,
1128 }
1129 testFacts = append(testFacts, tgf)
1130 }
1131 }
1132 }
1133 1134 var diags []Diagnostic
1135 for _, a := range root.deps {
1136 a := a.(*analyzerAction)
1137 diags = append(diags, a.Diagnostics...)
1138 }
1139 return analysisResult{
1140 facts: gobFacts,
1141 testFacts: testFacts,
1142 diagnostics: diags,
1143 unused: unusedResult,
1144 }, nil
1145 }
1146 1147 func registerGobTypes(analyzers []*analysis.Analyzer) {
1148 for _, a := range analyzers {
1149 for _, typ := range a.FactTypes {
1150 // FIXME(dh): use RegisterName so we can work around collisions
1151 // in names. For pointer-types, gob incorrectly qualifies
1152 // type names with the package name, not the import path.
1153 gob.Register(typ)
1154 }
1155 }
1156 }
1157 1158 func allAnalyzers(analyzers []*analysis.Analyzer) []*analysis.Analyzer {
1159 seen := map[*analysis.Analyzer]struct{}{}
1160 out := make([]*analysis.Analyzer, 0, len(analyzers))
1161 var dfs func(*analysis.Analyzer)
1162 dfs = func(a *analysis.Analyzer) {
1163 if _, ok := seen[a]; ok {
1164 return
1165 }
1166 seen[a] = struct{}{}
1167 out = append(out, a)
1168 for _, dep := range a.Requires {
1169 dfs(dep)
1170 }
1171 }
1172 for _, a := range analyzers {
1173 dfs(a)
1174 }
1175 return out
1176 }
1177 1178 // Run loads the packages specified by patterns, runs analyzers on
1179 // them and returns the results. Each result corresponds to a single
1180 // package. Results will be returned for all packages, including
1181 // dependencies. Errors specific to packages will be reported in the
1182 // respective results.
1183 //
1184 // If cfg is nil, a default config will be used. Otherwise, cfg will
1185 // be used, with the exception of the Mode field.
1186 func (r *Runner) Run(cfg *packages.Config, analyzers []*analysis.Analyzer, patterns []string) ([]Result, error) {
1187 analyzers = allAnalyzers(analyzers)
1188 registerGobTypes(analyzers)
1189 1190 r.Stats.setState(StateLoadPackageGraph)
1191 lpkgs, err := loader.Graph(r.cache, cfg, patterns...)
1192 if err != nil {
1193 return nil, err
1194 }
1195 r.Stats.setInitialPackages(len(lpkgs))
1196 1197 if len(lpkgs) == 0 {
1198 return nil, nil
1199 }
1200 1201 r.Stats.setState(StateBuildActionGraph)
1202 all := map[*loader.PackageSpec]*packageAction{}
1203 root := &packageAction{}
1204 for _, lpkg := range lpkgs {
1205 a := newPackageActionRoot(lpkg, all)
1206 root.deps = append(root.deps, a)
1207 a.triggers = append(a.triggers, root)
1208 }
1209 root.pending = uint32(len(root.deps))
1210 1211 queue := make(chan action)
1212 r.Stats.setTotalPackages(len(all) - 1)
1213 1214 r.Stats.setState(StateProcessing)
1215 go func() {
1216 for _, a := range all {
1217 if len(a.Deps()) == 0 {
1218 queue <- a
1219 }
1220 }
1221 }()
1222 1223 sr := newSubrunner(r, analyzers)
1224 for item := range queue {
1225 r.semaphore.Acquire()
1226 go genericHandle(item, root, queue, &r.semaphore, func(act action) error {
1227 return sr.do(act)
1228 })
1229 }
1230 1231 r.Stats.setState(StateFinalizing)
1232 out := make([]Result, 0, len(all))
1233 for _, item := range all {
1234 if item.Package == nil {
1235 continue
1236 }
1237 out = append(out, Result{
1238 Package: item.Package,
1239 Config: item.cfg,
1240 Initial: !item.factsOnly,
1241 Skipped: item.skipped,
1242 Failed: item.failed,
1243 Errors: item.errors,
1244 results: item.results,
1245 testData: item.testData,
1246 })
1247 }
1248 return out, nil
1249 }
1250