Fixes #12731.

I changed the code by intuition and I am still uncertain about lots of logic here. This initial commit can be compiled but increases the flakiness of at least grpclb_end2end test. And there is some asan issue I am looking at.

This change is 

Currently, grpclb will hand off the re-resolution decision to its embedded RR policy. This PR lets grpclb to be in charge of its re-resolution. The connectivity of the balancers will also be taken into consideration so that grpclb will re-resolve only when both balancers and backends have been unreachable for some timeout.

This change is 

~~This is VERY preliminary code. It builds, and I don't think it breaks any existing functionality, but that's about all I can say for it at this point. There are "FIXME" comments all over the place that need to be addressed before this is done, and there may yet be some issues that I have not yet identified.~~

~~That having been said, I am sending it out at this point just to get a sanity check as to whether or not this approach is even moderately reasonable. Please look at the high-level structure and let me know if it seems reasonable. Also, please look at my use of atomics (and some of the FIXME comments about things that aren't currently atomics but may need to be) and let me know if it looks sane.~~

~~Also, note that I came up with what I think is a better alternative to the tee_byte_stream implementation in #10448. It will need some additional work later to support hedging, but I think the current approach is fine for retries. If it looks good to you, I can probably revert that other PR.~~

~~I would welcome any and all suggestions you may have on this. Thanks!~~

This code is now ready for review!

I would appreciate review from the following folks:

• @ctiller: This code is really complex, and I am unwilling to trust it without careful review from you.
• @dgquintas: Please review from the perspective of your general knowledge of the client channel code.
• @ncteisen: Please review for compliance with the retry design.

Some things to note:

• This is a massive PR, so please let me know what I can do to make it easier to review. There is a large block comment in client_channel.cc describing the overall approach, but I'm sure it's not enough to understand everything. I would be happy to meet in person and do a code walk-through with some or all of you, or just answer any questions you may have.
• This includes all configurable retry functionality except for the stats, which I will work on in a separate PR. (That will require working with the Census folks to figure out the right way to expose these stats.)
• This does not include transparent retry support. That work depends on #11934 and will be added in a separate PR.
• This does not include hedging support. I've kept that use-case in mind while writing this code, but more design discussions will need to occur to figure out how to implement it without impacting performance.
• I have not yet verified that all tests pass, so there will no doubt be some fixes needed before this can be merged.
• I will be interested to see what the performance impact of this is. It should hopefully be not very much, since all of the synchronization overhead is taken care of by the call combiner code, which was already merged. There is some additional memory allocation needed for this, but that's mostly done on the arena, so it should hopefully not have a big impact. (And the additional memory allocations can be avoided by disabling retries via a channel arg.)

Please let me know if you have any questions. Thanks!

This change is 

Callers should properly initialize the memory.

Note that to avoid performance regressions we need some reordering in the initialization of grpc_call.

This behavior can be overridden using GRPC_ARENA_INIT_STRATEGY environment variable.

I had two more changes in the following files which I will skip in the PR because they are purely performance changes for cache coherency. I will upload that as a separate patch once this PR is merged: src/core/lib/channel/channel_stack.cc src/core/lib/surface/call.cc

This change is 

Implements the basic channel tracing architecture.

Adds channel_tracer and object_registry files. The tracers are held by channels and subchannels and only log channel creation for now. In a subsequent PR I will begin plumbing the tracing objects around and logging all of the interesting things happening

Picks up work from #10259. That PR was too stale to do a merge or rebase, but I want to maintain history

This change is 

This adds code to sort IP addresses resolved by grpc/c-ares wrapper per RFC 6724. This used Android's getaddrinfo as a guide (the differences between this and Android's implementation are mostly cosmetic and based around hooking the address-sorting code into the grpc code base). So, the code is added as a custom submodule which is under Android's license.

I'd like to have more end-to-end tests and cross-platform tests for this, (e.g. run the resolver_component_tests on IPv4-only machines, IPv6-only machines, machines with modified routing tables, Windows, etc.,and verify sorting results in those scenarios), but it looks like that involves a lot of setup, and so this tests by mocking libc functions.

This change is 

Added multipool config for large instances and standard instances, replacing the sentinel file for large instances. Also updated the RBE container to the image containing bazel 0.16.1

When Grpc.Tools is added to a C# project, the project recognizes .proto files. These .proto are compiled into .cs files placed in the object directory (under obj/...), and are not shown in Visual Studio, but are available to Intellisense nevertheless.

The scripts support both "classic" projects and the new dotnet style projects. In line with the latter ones (when .cs files are just dropped into the directory are picked up and compiled), .proto files are handled the same way, so just placing a .proto file into the folder is enough to compile it, and compile its protoc outputs into the project target assembly.

The "classic" project requires adding .proto files by one, and they are assigned a correct build action automatically. To manually add proto files to msbuild script, use 'ProtoBuf' item type.

For native code C++ projects, there is no added support beyond just dropping the compiler as before. For all other project types, an error message is generated on an attempted build. More C++ support is on the way.

Dependency change: the package now adds a dependency on Grpc.Tools of a matching version for managed projects only. This is naturally skipped for native C++, as the managed runtime is not required.

Also packaged are proto includes for well-known types.

Validated and confirmed working:

• Windows (VS, classic C#, dotnet C#, C++)
• Linux/Mono (both classic C# and dotnet C#)
• Linux/dotnet (dotnet C# project only)

There are quite a few questions I wanted to discuss. If anyone could help me with proper decicions, I would appreciate it.

1. The captialization of generated .cs files. When I compile fooBar.proto, the resulting .cs file is called FooBar.proto. Is there a way around that? This does not seem to make sense in the build like this, when the .cs files are treated at best as temporary. I can certainly script around the issue in msbuild scripts, but why have it in the first place?

2. If x.proto does not contain any RPC services. the corresponding xGrpc.cs is not generated (FWIW, the cpp plugin does not do that). This is not good for the build. While it is certainly possible to see if the file is there after compilation, it is impossible to learn if it was actually produced by it. In other words, if I take a service definition out of x.proto after having previously compiled it, I end up with a stale xGrpc.cs that is added to compilation. Of course, it is possible to write some msbuild incantations about this issue as well, but it would be much cleaner just to create a nearly empty xGrpc.cs stub even if no services are defined. So options I am asking about here are

• a) Leave it as is and script some very ugly transformations around it.
• b) Add a plugin option to do always generate the stub.
• c) In fact, forget the option and just always generate the stub.

3. I want to package cpp plugins also, and support to integrate gRPC into the build process. (I already have it packaged internally). This will cover the build process in Visual Studio. Should we do that?

4. I am adding a bunch of files to the root of src/chsarp that only go into the nuget package. Is it better to place them into a subdirectory? Is Grpc.Tools subdirectory good?

/cc @jskeet @hcoona
Closes #4805
/* #13098 */

If you liked internalization, you'll love C++ API review. The purpose of this PR is to clean up the use of breakable core structs from C++ public API. This has previously been done for the features that interfaced with code generation, but this now needs to be more pervasive. The basic principle is that core surface APIs and C++ implementation details should not leak through to C++ public API.

The implementation strategy is

1. when possible, privatize features that should have never been exposed
2. deprecate pieces if they can be easily and meaningfully substituted with a purely C++ API
3. co-opt items for which there is no other option, but try to give an alternative and more idiomatically C++ version for those when possible

This supercedes #14106 and will be matched to a gRFC soon.

This change is 

See Issue https://github.com/grpc/grpc/issues/17264

• Expose the C-Core API in Cython layer
• Handle the object translation
• Create a separate package for Channelz specifically
• Adding utilities for the new package

Background

gRPC spec defined two trailing data to indicate the status of a RPC call Status and Status-Message (Spec). But status code and a text message themselves are insufficient to express more complicate situation like the RPC call failed because of quota check of certain policy, or the stack trace from the crashed server, or obtain retry delay from the server (see error details).

Objective

So, gRPC needs a mechanism to transport those complex status. It is implemented in C++, Java, Golang, and now Python.

Design

It is quite straight forward, we provide two API:

• A server API to pack the rich status proto.
• A client API to unpack the rich status proto.

The transportation is done by setting/getting a trailing metadata entry named grpc-status-details-bin. Rich status message is encoded by ProtoBuf and transmit as binary.

Implementation

This PR

Related Issue

Addresses: https://github.com/grpc/grpc/issues/16366

What version of gRPC and what language are you using?

1.51.0,master

What operating system (Linux, Windows,...) and version?

Linux

What runtime / compiler are you using (e.g. python version or version of gcc)

C++ gcc11

What did you do?

use grpc proxyless in istio,but update routeconfig failed

What did you expect to see?

update routeconfig ok

What did you see instead?

1. After the program runs, there is the following error log and grpcdebug info.

[/data/grpc/src/core/ext/xds/xds_route_config.cc:1126][xds_client 0x448a9860] invalid RouteConfiguration outbound|50051||dev-nick.hlddz.svc.cluster.local: INVALID_ARGUMENT: errors validating RouteConfiguration resource: [field:virtual_hosts[2].routes error:no valid routes in VirtualHost; field:virtual_hosts[3].routes error:no valid routes in VirtualHost]

Name                                                   Status           Version                     Type                                                           LastUpdated
dev-meshgate:50051                                     ACKED            2023-01-05T08:07:45Z/1425   type.googleapis.com/envoy.config.listener.v3.Listener          12 seconds ago   
dev-nick:50051                                         ACKED            2023-01-05T07:40:47Z/1423   type.googleapis.com/envoy.config.listener.v3.Listener          22 minutes ago   
monitor:50051                                          DOES_NOT_EXIST                               type.googleapis.com/envoy.config.listener.v3.Listener                           
outbound|50051||dev-meshgate.hlddz.svc.cluster.local   NACKED                                       type.googleapis.com/envoy.config.route.v3.RouteConfiguration                    
outbound|50051||dev-nick.hlddz.svc.cluster.local       NACKED                                       type.googleapis.com/envoy.config.route.v3.RouteConfiguration   

2. After dumping the configuration file, it is found that the following configuration is used for normal http cgi routing

virtual_hosts {
  name: "inner.mcgitest.hlddz.huanle.qq.com:50051"
  domains: "inner.mcgitest.hlddz.huanle.qq.com"
  domains: "inner.mcgitest.hlddz.huanle.qq.com:50051"
  routes {
    match {
      prefix: "/cgi-bin/CommonMobileCGI/pandoraCgiProxy"
      case_sensitive {
        value: true
      }
    }
    route {
      cluster: "outbound|50051|online|dev-pandoracgiproxy.hlddz.svc.cluster.local"
      timeout {
      }
      retry_policy {
        retry_on: "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes"
        num_retries {
          value: 2
        }
        retry_host_predicate {
          name: "envoy.retry_host_predicates.previous_hosts"
        }
        host_selection_retry_max_attempts: 5
        retriable_status_codes: 503
      }
      hash_policy {
        header {
          header_name: "routekey"
        }
      }
      max_stream_duration {
        max_stream_duration {
        }
      }
    }
    metadata {
      filter_metadata {
        key: "istio"
        value {
          fields {
            key: "config"
            value {
              string_value: "/apis/networking.istio.io/v1alpha3/namespaces/hlddz/virtual-service/dev-pandoracgiproxy"
            }
          }
        }
      }
    }
    decorator {
      operation: "dev-pandoracgiproxy.hlddz.svc.cluster.local:50051/cgi-bin/CommonMobileCGI/pandoraCgiProxy*"
    }
  }
  include_request_attempt_count: true
}

3. After reading the source code, I found the following logic. When parsing the Routerconfiguration, the match field will be validated. When the configuration contains entries that do not conform to the grpc specification, the entire configuration update will be discarded. The source code is as follows

 const envoy_config_route_v3_Route* const* routes =
        envoy_config_route_v3_VirtualHost_routes(virtual_hosts[i], &num_routes);
    for (size_t j = 0; j < num_routes; ++j) {
      ValidationErrors::ScopedField field(errors, absl::StrCat("[", j, "]"));
      auto route = ParseRoute(context, routes[j], virtual_host_retry_policy,
                              rds_update.cluster_specifier_plugin_map,
                              &cluster_specifier_plugins_not_seen, errors);
      if (route.has_value()) vhost.routes.emplace_back(std::move(*route));
    }
    if (errors->size() == original_error_size && vhost.routes.empty()) {
      errors->AddError("no valid routes in VirtualHost");
    }

if (envoy_config_route_v3_RouteMatch_has_prefix(match)) {
    absl::string_view prefix =
        UpbStringToAbsl(envoy_config_route_v3_RouteMatch_prefix(match));
    // For any prefix that cannot match a path of the form "/service/method",
    // ignore the route.
    if (!prefix.empty()) {
      // Does not start with a slash.
      if (prefix[0] != '/') return absl::nullopt;
      std::vector<absl::string_view> prefix_elements =
          absl::StrSplit(prefix.substr(1), absl::MaxSplits('/', 2));
      // More than 2 slashes.
      if (prefix_elements.size() > 2) return absl::nullopt;
      // Two consecutive slashes.
      if (prefix_elements.size() == 2 && prefix_elements[0].empty()) {
        return absl::nullopt;
      }
    }
    type = StringMatcher::Type::kPrefix;
    match_string = std::string(prefix);
  } 

4. Our service runs in istio, which is a large heterogeneous system. There are grpc services and regular http cgi services, so the routing configuration may not conform to the grpc specification. Is it possible to just skip this piece of configuration when reading a configuration that does not meet the specifications, instead of discarding the entire configuration update, similar modifications are as follows

for (size_t i = 0; i < num_virtual_hosts; ++i) {
    ...
    const envoy_config_route_v3_Route* const* routes =
        envoy_config_route_v3_VirtualHost_routes(virtual_hosts[i], &num_routes);
    for (size_t j = 0; j < num_routes; ++j) {
      ValidationErrors::ScopedField field(errors, absl::StrCat("[", j, "]"));
      auto route = ParseRoute(context, routes[j], virtual_host_retry_policy,
                              rds_update.cluster_specifier_plugin_map,
                              &cluster_specifier_plugins_not_seen, errors);
      if (route.has_value()) vhost.routes.emplace_back(std::move(*route));
    }
    if (errors->size() == original_error_size && vhost.routes.empty()) {
      // ignore this invalid virtualhost update
      continue;
    }
  }

5. After I followed the above modification method, it worked well in my system. I think it may be more friendly to implement it in a heterogeneous operating environment. grpcdebug info

Name                                                         Status    Version                     Type                                                                 LastUpdated
outbound|50051|online|dev-meshgate.hlddz.svc.cluster.local   ACKED     2023-01-05T08:07:45Z/1425   type.googleapis.com/envoy.config.cluster.v3.Cluster                  13 seconds ago   
outbound|50051|online|dev-nick.hlddz.svc.cluster.local       ACKED     2023-01-05T08:07:45Z/1425   type.googleapis.com/envoy.config.cluster.v3.Cluster                  15 seconds ago   
outbound|50051|online|dev-meshgate.hlddz.svc.cluster.local   ACKED     2023-01-05T08:07:45Z/1425   type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment   13 seconds ago   
outbound|50051|online|dev-nick.hlddz.svc.cluster.local       ACKED     2023-01-05T08:07:45Z/1425   type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment   15 seconds ago   
dev-meshgate:50051                                           ACKED     2023-01-05T08:07:45Z/1425   type.googleapis.com/envoy.config.listener.v3.Listener                13 seconds ago   
dev-nick:50051                                               ACKED     2023-01-05T08:07:45Z/1425   type.googleapis.com/envoy.config.listener.v3.Listener                15 seconds ago   
outbound|50051||dev-meshgate.hlddz.svc.cluster.local         ACKED     2023-01-05T08:07:45Z/1425   type.googleapis.com/envoy.config.route.v3.RouteConfiguration         13 seconds ago   
outbound|50051||dev-nick.hlddz.svc.cluster.local             ACKED     2023-01-05T08:07:45Z/1425   type.googleapis.com/envoy.config.route.v3.RouteConfiguration         13 seconds ago 

Anything else we should know about your project / environment?

Both 1.51.x and the latest master have the above-mentioned problems, and the above-mentioned source code is the code of the latest master referenced

iomgr sets the grpc_tcp_client_impl vtable here as part of the first grpc_init which overwrites the fuzz_tcp_client_vtable set by api_fuzzer. This caused inconsistent behavior between whether api_fuzzer is used to run one test case or multiple test cases (since later test case run will overwrite the vtable again and use the intended one).

The grpc bazel rules will not be compatible with future versions of bazel due to
--incompatible_disable_starlark_host_transitions

This PR follows the recommended migration.

A workaround for https://github.com/grpc/grpc/issues/31975 - use x86 protoc binaries on Windows arm64.

Unable to test as I don't have access to Windows-arm64.

Also not sure how NUnit PlatformAttribute handles arm64 (could not find any information). It is possible that tests will fail if run on Windows-arm64 with full .NET framework if NUnit doesn't distinguish between x64 and arm64 (just identifies a 64 bit platform). Not sure that we run tests on Windows-arm64.

Followup for https://github.com/grpc/grpc/pull/31555.

objc CFStream tests now run under bazel.

• there is no need for the complexity of "ios-test-cfstream-tests" under run_tests.py
• ios-test-cfstream-tests is the last test task under the grpc_basictests_objc_ios test job, which means we'll be able to get rid of the job completely (the job is slow so this is welcome)
• right now the actual test implementation in CFStreamClientTests.mm and CFStreamEndpointTests.mm is duplicated (there are 2 almost identical copies of each of the files in the repository).