DineIQ Analytics was designed around this problem. The project is a web-based restaurant intelligence platform that combines Big Data processing, data science, machine learning, and interactive analytics. Its purpose is to move restaurant analysis beyond basic charts and toward evidence-based conclusions about menu performance, customer behavior, demand, wastage, pricing, promotions, and operational risks.

The Software Requirements Specification describes DineIQ as a "Data Science Intelligence Arena" based on the MenuMatrix Dining Intelligence theme. It specifically requires the application to analyze large restaurant datasets and provide business insights, predictive results, recommendations, and comparisons between independent Spark and Python analytical pipelines.

The Business Problem

Traditional restaurant reporting often focuses on questions such as how much was sold or which item generated the most revenue. Those numbers are useful, but they can hide important relationships. A dish may have high sales while producing a weak margin. Another item may sell less frequently but generate strong contribution margin and repeat purchases. A promotion may increase order volume while simultaneously reducing profitability or increasing wastage.

DineIQ approaches the problem from multiple dimensions. Menu analysis considers sales volume, revenue, cost, contribution margin, profitability, ratings, repeat purchases, wastage, promotion dependency, and trends. Customer analysis considers behavior over time rather than treating every customer as identical. Forecasting uses historical patterns to estimate future demand, while anomaly detection looks for unusual sales, ratings, discounts, and transactions.

This broader approach follows the SRS requirement that high sales volume alone must not determine whether a menu item is successful. The specification also requires difficult cases such as high-selling loss-making dishes, highly profitable but rarely purchased dishes, popular dishes with excessive wastage, and items that behave differently across locations.

From Restaurant Data to Big Data Processing

A central design decision in DineIQ is the use of Apache Spark and PySpark for large-scale processing. The backend defines explicit Spark schemas for customers, ingredients, inventory, menu categories, menu items, menu-item ingredients, orders, order items, pricing history, promotions, ratings, restaurant menus, restaurants, and wastage.

The ingestion layer reads the restaurant CSV files using predefined schemas. It also demonstrates schema inference separately and checks partition counts for order-item data. This is important because the SRS does not treat Big Data as simply reading a large CSV. It explicitly requires schema definition, schema inference, data-type validation, large-file loading, multiple-file ingestion, and partition handling.

The project therefore separates ingestion from later processing stages. This makes the pipeline easier to validate and allows data problems to be identified before they affect analytical results.

Data Quality Before Intelligence

Reliable analytics starts with reliable data. DineIQ includes a dedicated data-quality stage rather than assuming that generated or imported records are perfect.

The quality pipeline checks for problems such as null values and duplicate identifiers and is structured around the broader validation requirements in the SRS. The specification calls for detection of missing values, duplicate orders and order lines, invalid prices, negative quantities, invalid dates and ratings, missing identifiers, impossible wastage quantities, incorrect discounts, cancelled transactions, inconsistent units, and invalid location references. A Data Quality Report is also required.

After assessment, the preprocessing layer cleans or manages problematic records according to defined rules. This distinction is important: data cleaning is not treated as an invisible operation. The SRS requires cleaning decisions to be recorded, helping make the analytical pipeline more understandable and auditable.

Integrating the Restaurant Data Model

Restaurant intelligence becomes much more powerful when individual datasets can be connected. Orders can be related to customers and order items; order items can be connected to menu items; menu items can be connected to categories, pricing history, ratings, inventory, and wastage. Orders can also be connected to restaurants and promotions.

DineIQ contains a dedicated Spark integration stage for these relationships. This provides the foundation for downstream feature engineering and analytics. Instead of analyzing an isolated sales table, the application can connect operational and behavioral information into a richer analytical view.

The SRS specifically requires these relationships and expects Spark SQL or PySpark joins to perform the integration. It also requires at least one processed Big Data dataset to be stored in Parquet format, demonstrating an efficient storage format for processed analytical data.

Feature Engineering: Turning Records into Signals

Raw restaurant records are rarely the final inputs for analytics. DineIQ therefore includes a feature-engineering stage that derives business-oriented measurements.

Examples include item revenue, cost, contribution margin, profit percentage, order frequency, popularity, repeat-purchase rate, average rating, rating trend, wastage percentage, promotion dependency, discount percentage, customer recency, frequency, monetary value, average order value, peak-hour frequency, weekend-order ratio, location performance, channel preference, basket size, and price-change percentage.

These features transform transactional data into signals that can be used by both descriptive analytics and machine-learning models. The SRS identifies these types of derived features as an explicit project requirement.

Menu Intelligence

One of DineIQ's major analytical areas is menu intelligence. The application separates profitability analysis from menu classification so that classification can be based on multiple indicators rather than one hard-coded field.

Menu items are classified into four conceptual groups: Profit Driver, Volume Driver, Hidden Opportunity, and Low Performer. A Profit Driver combines high demand and high profitability with acceptable wastage. A Volume Driver has strong demand but comparatively lower profitability. A Hidden Opportunity can have good profitability, ratings, or repeat purchasing while suffering from comparatively low visibility or sales. A Low Performer combines weak demand, weak profitability, excessive wastage, poor ratings, or other unfavorable indicators.

The backend also contains logic for slow-moving dishes and difficult menu cases. This matters because real restaurant decisions are rarely binary. A popular item can still be a poor business choice if its margin is weak or its wastage is excessive.

Customer Intelligence and RFM

DineIQ also looks at the restaurant from the customer's perspective. Customer segmentation is based on behavioral information such as recency, frequency, monetary value, average order value, visit frequency, favorite categories, promotion sensitivity, ordering channel, time-of-day preference, and repeat behavior.

RFM analysis provides a compact way to understand three important dimensions of customer value: how recently a customer purchased, how frequently they purchase, and how much they spend. The project implements a dedicated RFM analysis module and presents segmentation, RFM, and churn-risk views through the web interface.

The SRS recommends segments such as High-Value Loyal Customers, Frequent Customers, Promotion-Driven Customers, At-Risk Customers, New Customers, and Occasional Customers. It also requires RFM calculations to be performed independently or used as an input to segmentation.

The churn-risk component is deliberately presented as a rule-based indicator rather than being described as a probability. This is an example of how analytical output can be communicated with appropriate limitations instead of giving users a misleading sense of certainty.

Market-Basket Analysis and Cross-Selling

A restaurant does not only need to know which individual products sell. It also needs to understand which products appear together in the same baskets.

DineIQ includes market-basket analysis designed to identify frequently purchased combinations and calculate association-rule metrics such as support, confidence, and lift. These results can feed bundle, cross-sell, and upsell recommendations.

This follows the SRS requirement that bundle recommendations must be supported by association-rule evidence rather than simply being arbitrary product pairings.

Forecasting and Time-Aware Validation

Demand forecasting is another important part of the platform. DineIQ includes forecasting modules for menu items, categories, restaurant locations, and time periods, together with forecast-accuracy evaluation and time-aware validation.

Time-aware validation is particularly important for restaurant forecasting. Randomly mixing old and future observations can allow future information to influence training and produce misleading results. The SRS therefore requires earlier periods to be used for training and later unseen periods for testing, with future information excluded from training features. Forecasts can then be evaluated using measures such as MAE, RMSE, MAPE, and R² where appropriate.

The project also exposes forecast results through a dedicated Forecast page, allowing historical demand, predicted demand, actual-versus-predicted values, and forecast errors to be presented as part of the application.

Wastage, Pricing, and Promotion Intelligence

Food wastage is treated as an analytical problem rather than simply an inventory record. DineIQ includes wastage analysis and a separate wastage-risk prediction pipeline. The analysis can consider menu items, categories, locations, dates, demand, inventory consumption, promotions, and preparation quantities.

Pricing intelligence examines the relationship between price, demand, revenue, contribution margin, discounts, ratings, and repeat purchasing. Price sensitivity can then be used to identify items whose demand responds more strongly to price changes.

Promotion intelligence follows a similar philosophy. A promotion is not automatically considered successful because sales increased. The SRS requires evaluation across order volume, revenue, contribution margin, customer acquisition, repeat purchases, average order value, wastage, and post-promotion behavior. It also specifically calls for detection of "promotion traps," such as promotions that increase sales while reducing profit.

Detecting Anomalies

Restaurants can also experience unusual events that deserve investigation. DineIQ contains separate detection modules for rating anomalies and sales anomalies.

Examples include sudden rating spikes or drops, excessive identical ratings, unusually large numbers of ratings in short periods, sudden sales changes, abnormal order values, unusual discounts, unexpected demand, and duplicate transactions.

Anomaly detection does not automatically prove that an event is fraudulent or incorrect. Instead, it creates an analytical signal that can direct attention toward records or periods requiring investigation. This distinction is important when designing responsible analytics.

Two Independent Data Science Pipelines

One of the most technically distinctive parts of DineIQ is the dual-pipeline requirement. The SRS requires a Big Data pipeline using Apache Spark, PySpark, Spark SQL, and Spark MLlib, alongside an independent Python Data Science pipeline using tools such as Pandas and scikit-learn. Both pipelines must solve selected analytical problems independently using equivalent underlying records.

For menu classification, the project contains a Spark MLlib implementation and a separate Python implementation. The comparison pipeline generates out-of-fold predictions and then compares the outputs by record.

The comparison includes the actual class, Spark prediction, Python prediction, match status, confidence or probability information where available, probability difference, correctness indicators, consistency status, model versions, and explanations for disagreements. This makes model disagreement an analytical artifact rather than something hidden from the user.

The SRS explicitly states that exact equality is not required for independently trained models. Instead, the application should document matches, differences, and the overall agreement percentage.

Recommendation Engine and What-If Analysis

Analytics become more useful when they lead to actions. DineIQ therefore contains a recommendation engine with separate components for recommendation generation, evidence, priority, and bundle recommendations.

The SRS requires recommendations to display the analytical evidence supporting them. For example, an item could be recommended for promotion because it has strong contribution margin, a strong rating, low wastage, low current order frequency, and good repeat purchasing among existing buyers. Recommendations can also address preparation quantities, pricing, bundles, low performers, inventory timing, customer targeting, promotions, and anomalies.

The platform also includes what-if scenario analysis. Users can simulate changes such as increasing or decreasing prices, changing discount percentages, increasing promotion frequency, removing a menu item, changing preparation quantity, or modifying demand and wastage assumptions. The resulting revenue, contribution margin, demand, wastage, and profitability values are presented as estimates rather than actual historical results.

Web Application Architecture

The application is built with a FastAPI backend and a React frontend. The backend uses SQLModel with PostgreSQL or SQLite support, while the frontend uses React, Vite, and Recharts.

The React interface is divided into focused pages for areas including Executive analytics, Menu intelligence, Customers, Basket analysis, Forecasting, Wastage, Pricing, Promotions, Ratings, Locations, Peak periods, Recommendations, Reports, What-If analysis, Jobs, and Dual-Pipeline comparison.

This organization mirrors the analytical structure of the backend. Instead of presenting one overloaded dashboard, the application gives each business question its own analytical surface. The Executive page provides a high-level view, while specialized pages allow users to investigate individual areas in greater depth.

The SRS also requires an executive dashboard, menu dashboard, customer dashboard, wastage dashboard, forecast dashboard, and dual-pipeline comparison dashboard.

Testing, Validation, and Project Integrity

DineIQ includes validation modules throughout the backend. Many analytical components have corresponding validation scripts, while the repository also contains application tests and permission checks.

This approach is especially important because the competition requirements emphasize explainability and genuine implementation. The SRS requires meaningful commits, a development log, understandable code, testing, and the ability for team members to explain their modules. It also warns against unused code, unexplained code, fabricated metrics, hard-coded insights, and external decision APIs.

The project documentation therefore needs to be treated as part of the engineering work, not an afterthought. Model versions, processing stages, analytical assumptions, testing results, limitations, and AI-tool usage should remain traceable.

Challenges and Lessons Learned

The biggest lesson from building a system like DineIQ is that restaurant intelligence is an integration problem as much as a machine-learning problem. A sophisticated model cannot compensate for disconnected records, invalid identifiers, inconsistent data, or poorly defined business features.

Another challenge is keeping two analytical pipelines genuinely independent. If one pipeline simply receives predictions from the other, the comparison loses its value. DineIQ's comparison workflow instead builds Spark and Python predictions independently and then evaluates their consistency.

A third challenge is translating technical outputs into business language. Restaurant managers do not necessarily need to see every transformation or model parameter. They need to understand what changed, why it matters, how strong the evidence is, and what action can be considered. That is why DineIQ combines analytical modules with evidence-backed recommendations and dedicated dashboard views.

Performance, Security, and Limitations

The SRS defines performance, scalability, usability, accuracy, and availability targets. It expects the architecture to support growth to at least five million order-line records without a complete redesign and sets model-performance targets such as at least 85% test accuracy or a macro F1-score of at least 0.80 where appropriate.

At the same time, the project's README identifies practical limits. Saved forecasts currently support 1 to 30 days, the independent Spark/Python comparison is implemented for menu classification, analytics are read from saved pipeline outputs, and adding a managed restaurant location requires a pipeline rerun before analytical results are available.

These limitations are not weaknesses to hide. Documenting them makes the system easier to understand, test, and improve.

Future Enhancements

Future development could extend the platform with additional independent machine-learning tasks, richer live job monitoring, broader forecasting strategies, more sophisticated customer models, and deeper integration with operational systems.

The SRS currently keeps live POS systems, payment gateways, third-party delivery platforms, and enterprise restaurant-management integrations outside the project scope.

A future production-oriented version could therefore connect DineIQ to live restaurant systems while preserving the existing analytical architecture. Additional model versioning, automated retraining, stronger monitoring, and expanded scenario simulation could also make the platform more suitable for continuous operational use.

Conclusion

DineIQ Analytics demonstrates how restaurant data can be transformed into a complete intelligence workflow. The project begins with interconnected operational records, validates and cleans those records, integrates them with Spark, engineers analytical features, performs business analysis, trains independent machine-learning pipelines, compares model outputs, and presents the results through a web application.

More importantly, the project is designed around a principle that restaurant analytics should explain more than what happened. It should help explain why it happened, where it happened, what may happen next, and what evidence supports a possible response.

By combining Apache Spark, PySpark, Spark SQL, Parquet, Spark MLlib, Python data science, FastAPI, React, and business-focused analytical modules, DineIQ turns a large restaurant dataset into a structured intelligence platform. Its menu intelligence, customer segmentation, RFM analysis, market-basket analysis, forecasting, wastage prediction, pricing analysis, promotion analysis, anomaly detection, recommendations, and what-if scenarios create a foundation for data-driven restaurant decision-making.

The result is not simply a collection of charts. It is an end-to-end analytical system built to connect data engineering, machine learning, and business intelligence in one application.