We’re rewarding the question askers & reputations are being recalculated! Read more.
3 replaced http://stackoverflow.com/ with https://stackoverflow.com/
source | link

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria:

All married males in NY or LA assign to Bucket/Segment 1;

Currently at the end of day, I query users from a TB scale table in cloud, one bucket criteria at a time, based on thousands of different bucket criterias. Then generate a daily feed for each of these segments.

I use to query the database every night but the number of segments/buckets have become so huge that its no longer is feasible to batch query each bucket criteria every night and populate buckets.

Instead I am thinking of moving to streaming architecture where a user is assigned to the segment as they come in. For that I am looking to load segment definitions into a directional (multi edge) graph data structure and determine which user qualifies for which segment as the user come in. Sample Graph below: enter image description here

In case you are wondering why do I need multiple edges between two vertices, it is to represent and condition per bucket criteria. For e.g. if Segment 1 is

All married males in NY or LA assign to Bucket/Segment 1;

And Segment 2 is:

All married males assign to Bucket/Segment 2;

if Segment 1 and 2 share the same edge then that will be a problem because Segment 1 results are just a subset of Segment 2 results. Segment 2 has no city filter.

This graph theoretically represents all the relations for my use perfectly, but the problem is after implementing this graph the performance of graph creation algorithm has been disappointing. Most the time is spent creating edges for vertices in a bucket criteria for all possible paths to that edge.

Some filter values like Zip Codes can have tens of thousands of values which results in a proliferation of edges between nodes. I am looking for a way to optimize this graph creation.

Edit: Providing more details as per answer below;

  1. Technology I am using: Google Cloud Dataflow (Managed Apache Beam). More details on how I am implementing this are in the comment herehere
  2. Clean rules set to use sets instead of actual values: I am using sets for alot of rules like zipcodes and have it implemented already as zip in { "98500","98800",.... }. But for some rules for e.g. user email you can't really have a set and to represent each user email in the graph as a vertex.
  3. Implement truth propagation graph: Will look more into this currently, I am using HashMap for my vertex and edges. My vertex is v={vertex_id, HashMap} and my edge = {edge_id,edge_to_vertex}. And my graph is implemented as HashMap

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria:

All married males in NY or LA assign to Bucket/Segment 1;

Currently at the end of day, I query users from a TB scale table in cloud, one bucket criteria at a time, based on thousands of different bucket criterias. Then generate a daily feed for each of these segments.

I use to query the database every night but the number of segments/buckets have become so huge that its no longer is feasible to batch query each bucket criteria every night and populate buckets.

Instead I am thinking of moving to streaming architecture where a user is assigned to the segment as they come in. For that I am looking to load segment definitions into a directional (multi edge) graph data structure and determine which user qualifies for which segment as the user come in. Sample Graph below: enter image description here

In case you are wondering why do I need multiple edges between two vertices, it is to represent and condition per bucket criteria. For e.g. if Segment 1 is

All married males in NY or LA assign to Bucket/Segment 1;

And Segment 2 is:

All married males assign to Bucket/Segment 2;

if Segment 1 and 2 share the same edge then that will be a problem because Segment 1 results are just a subset of Segment 2 results. Segment 2 has no city filter.

This graph theoretically represents all the relations for my use perfectly, but the problem is after implementing this graph the performance of graph creation algorithm has been disappointing. Most the time is spent creating edges for vertices in a bucket criteria for all possible paths to that edge.

Some filter values like Zip Codes can have tens of thousands of values which results in a proliferation of edges between nodes. I am looking for a way to optimize this graph creation.

Edit: Providing more details as per answer below;

  1. Technology I am using: Google Cloud Dataflow (Managed Apache Beam). More details on how I am implementing this are in the comment here
  2. Clean rules set to use sets instead of actual values: I am using sets for alot of rules like zipcodes and have it implemented already as zip in { "98500","98800",.... }. But for some rules for e.g. user email you can't really have a set and to represent each user email in the graph as a vertex.
  3. Implement truth propagation graph: Will look more into this currently, I am using HashMap for my vertex and edges. My vertex is v={vertex_id, HashMap} and my edge = {edge_id,edge_to_vertex}. And my graph is implemented as HashMap

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria:

All married males in NY or LA assign to Bucket/Segment 1;

Currently at the end of day, I query users from a TB scale table in cloud, one bucket criteria at a time, based on thousands of different bucket criterias. Then generate a daily feed for each of these segments.

I use to query the database every night but the number of segments/buckets have become so huge that its no longer is feasible to batch query each bucket criteria every night and populate buckets.

Instead I am thinking of moving to streaming architecture where a user is assigned to the segment as they come in. For that I am looking to load segment definitions into a directional (multi edge) graph data structure and determine which user qualifies for which segment as the user come in. Sample Graph below: enter image description here

In case you are wondering why do I need multiple edges between two vertices, it is to represent and condition per bucket criteria. For e.g. if Segment 1 is

All married males in NY or LA assign to Bucket/Segment 1;

And Segment 2 is:

All married males assign to Bucket/Segment 2;

if Segment 1 and 2 share the same edge then that will be a problem because Segment 1 results are just a subset of Segment 2 results. Segment 2 has no city filter.

This graph theoretically represents all the relations for my use perfectly, but the problem is after implementing this graph the performance of graph creation algorithm has been disappointing. Most the time is spent creating edges for vertices in a bucket criteria for all possible paths to that edge.

Some filter values like Zip Codes can have tens of thousands of values which results in a proliferation of edges between nodes. I am looking for a way to optimize this graph creation.

Edit: Providing more details as per answer below;

  1. Technology I am using: Google Cloud Dataflow (Managed Apache Beam). More details on how I am implementing this are in the comment here
  2. Clean rules set to use sets instead of actual values: I am using sets for alot of rules like zipcodes and have it implemented already as zip in { "98500","98800",.... }. But for some rules for e.g. user email you can't really have a set and to represent each user email in the graph as a vertex.
  3. Implement truth propagation graph: Will look more into this currently, I am using HashMap for my vertex and edges. My vertex is v={vertex_id, HashMap} and my edge = {edge_id,edge_to_vertex}. And my graph is implemented as HashMap
2 added 939 characters in body
source | link

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria:

All married males in NY or LA assign to Bucket/Segment 1;

Currently at the end of day, I query users from a TB scale table in cloud, one bucket criteria at a time, based on thousands of different bucket criterias. Then generate a daily feed for each of these segments.

I use to query the database every night but the number of segments/buckets have become so huge that its no longer is feasible to batch query each bucket criteria every night and populate buckets.

Instead I am thinking of moving to streaming architecture where a user is assigned to the segment as they come in. For that I am looking to load segment definitions into a directional (multi edge) graph data structure and determine which user qualifies for which segment as the user come in. Sample Graph below: enter image description here

In case you are wondering why do I need multiple edges between two vertices, it is to represent and condition per bucket criteria. For e.g. if Segment 1 is

All married males in NY or LA assign to Bucket/Segment 1;

And Segment 2 is:

All married males assign to Bucket/Segment 2;

if Segment 1 and 2 share the same edge then that will be a problem because Segment 1 results are just a subset of Segment 2 results. Segment 2 has no city filter.

This graph theoretically represents all the relations for my use perfectly, but the problem is after implementing this graph the performance of graph creation algorithm has been disappointing. Most the time is spent creating edges for vertices in a bucket criteria for all possible paths to that edge.

Some filter values like Zip Codes can have tens of thousands of values which results in a proliferation of edges between nodes. I am looking for a way to optimize this graph creation.

Edit: Providing more details as per answer below;

  1. Technology I am using: Google Cloud Dataflow (Managed Apache Beam). More details on how I am implementing this are in the comment here
  2. Clean rules set to use sets instead of actual values: I am using sets for alot of rules like zipcodes and have it implemented already as zip in { "98500","98800",.... }. But for some rules for e.g. user email you can't really have a set and to represent each user email in the graph as a vertex.
  3. Implement truth propagation graph: Will look more into this currently, I am using HashMap for my vertex and edges. My vertex is v={vertex_id, HashMap} and my edge = {edge_id,edge_to_vertex}. And my graph is implemented as HashMap

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria:

All married males in NY or LA assign to Bucket/Segment 1;

Currently at the end of day, I query users from a TB scale table in cloud, one bucket criteria at a time, based on thousands of different bucket criterias. Then generate a daily feed for each of these segments.

I use to query the database every night but the number of segments/buckets have become so huge that its no longer is feasible to batch query each bucket criteria every night and populate buckets.

Instead I am thinking of moving to streaming architecture where a user is assigned to the segment as they come in. For that I am looking to load segment definitions into a directional (multi edge) graph data structure and determine which user qualifies for which segment as the user come in. Sample Graph below: enter image description here

In case you are wondering why do I need multiple edges between two vertices, it is to represent and condition per bucket criteria. For e.g. if Segment 1 is

All married males in NY or LA assign to Bucket/Segment 1;

And Segment 2 is:

All married males assign to Bucket/Segment 2;

if Segment 1 and 2 share the same edge then that will be a problem because Segment 1 results are just a subset of Segment 2 results. Segment 2 has no city filter.

This graph theoretically represents all the relations for my use perfectly, but the problem is after implementing this graph the performance of graph creation algorithm has been disappointing. Most the time is spent creating edges for vertices in a bucket criteria for all possible paths to that edge.

Some filter values like Zip Codes can have tens of thousands of values which results in a proliferation of edges between nodes. I am looking for a way to optimize this graph creation.

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria:

All married males in NY or LA assign to Bucket/Segment 1;

Currently at the end of day, I query users from a TB scale table in cloud, one bucket criteria at a time, based on thousands of different bucket criterias. Then generate a daily feed for each of these segments.

I use to query the database every night but the number of segments/buckets have become so huge that its no longer is feasible to batch query each bucket criteria every night and populate buckets.

Instead I am thinking of moving to streaming architecture where a user is assigned to the segment as they come in. For that I am looking to load segment definitions into a directional (multi edge) graph data structure and determine which user qualifies for which segment as the user come in. Sample Graph below: enter image description here

In case you are wondering why do I need multiple edges between two vertices, it is to represent and condition per bucket criteria. For e.g. if Segment 1 is

All married males in NY or LA assign to Bucket/Segment 1;

And Segment 2 is:

All married males assign to Bucket/Segment 2;

if Segment 1 and 2 share the same edge then that will be a problem because Segment 1 results are just a subset of Segment 2 results. Segment 2 has no city filter.

This graph theoretically represents all the relations for my use perfectly, but the problem is after implementing this graph the performance of graph creation algorithm has been disappointing. Most the time is spent creating edges for vertices in a bucket criteria for all possible paths to that edge.

Some filter values like Zip Codes can have tens of thousands of values which results in a proliferation of edges between nodes. I am looking for a way to optimize this graph creation.

Edit: Providing more details as per answer below;

  1. Technology I am using: Google Cloud Dataflow (Managed Apache Beam). More details on how I am implementing this are in the comment here
  2. Clean rules set to use sets instead of actual values: I am using sets for alot of rules like zipcodes and have it implemented already as zip in { "98500","98800",.... }. But for some rules for e.g. user email you can't really have a set and to represent each user email in the graph as a vertex.
  3. Implement truth propagation graph: Will look more into this currently, I am using HashMap for my vertex and edges. My vertex is v={vertex_id, HashMap} and my edge = {edge_id,edge_to_vertex}. And my graph is implemented as HashMap
1
source | link

Moving from Batch to Streaming architecture using a Graph datastructure

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria:

All married males in NY or LA assign to Bucket/Segment 1;

Currently at the end of day, I query users from a TB scale table in cloud, one bucket criteria at a time, based on thousands of different bucket criterias. Then generate a daily feed for each of these segments.

I use to query the database every night but the number of segments/buckets have become so huge that its no longer is feasible to batch query each bucket criteria every night and populate buckets.

Instead I am thinking of moving to streaming architecture where a user is assigned to the segment as they come in. For that I am looking to load segment definitions into a directional (multi edge) graph data structure and determine which user qualifies for which segment as the user come in. Sample Graph below: enter image description here

In case you are wondering why do I need multiple edges between two vertices, it is to represent and condition per bucket criteria. For e.g. if Segment 1 is

All married males in NY or LA assign to Bucket/Segment 1;

And Segment 2 is:

All married males assign to Bucket/Segment 2;

if Segment 1 and 2 share the same edge then that will be a problem because Segment 1 results are just a subset of Segment 2 results. Segment 2 has no city filter.

This graph theoretically represents all the relations for my use perfectly, but the problem is after implementing this graph the performance of graph creation algorithm has been disappointing. Most the time is spent creating edges for vertices in a bucket criteria for all possible paths to that edge.

Some filter values like Zip Codes can have tens of thousands of values which results in a proliferation of edges between nodes. I am looking for a way to optimize this graph creation.