Site Log Management: Collecting and Analyzing Access, Error, and Application Logs
1. Each Type of Log Handles One Job — Don’t Read Them Mixed Together
Many people treat every .log file on the server as “the logs” and dump them all together, then dig through them one by one when something goes wrong. In fact, the three types of logs answer three completely different questions. Access logs (Nginx/Apache access.log) answer “who requested what, and when”; error logs (Nginx error.log, PHP-FPM errors) answer “where did the server report an error”; application logs (the logs you write yourself in your code) answer “how far did this order get processed, and why did it fail”.
Here’s the counterintuitive judgment:When a production incident happens, the last thing you should dig through first is the access log. Access logs are high-volume and noisy — looking at them first means you’ll be buried under thousands of 200s. The correct order is to check error logs first to locate the point of failure, then use access logs to see the scope of impact (how many users, which URLs), and finally use application logs to reconstruct the business process. This is the same approach as AI Crawler Log Analysis emphasizing “first figure out who’s reading you”: classify first, then analyze.
| Log Type | Questions answered | Typical Location | Suggested Retention |
|---|---|---|---|
| Access Log | Who, when, and what was requested | Nginx access.log | 90 days hot data |
| Error Log | Where the server reported an error, and why | Nginx error.log / PHP errors | 1 year |
| Application Log | How far the business process got, and where it got stuck | Custom log in code | 30 days |
| System Log | Events on the server itself | /var/log/messages | 30 days |
2. Collection Pipeline: From “Scattered Across Files” to “Queryable in One Place”
The common state for a solo team is: one log for Nginx, one log for PHP, and one log the application writes itself, scattered across different directories. Once the server restarts or you switch machines, the history is cut off. The first step in log management isn’t analysis — it’s centralizing them.
A log pipeline has four stages: your app writes the file, the file gets rotated daily, logs are centralized in one place, and alerts fire on errors. For rotation, use logrotate to rotate daily and gzip the results, so you never end up with a multi-gigabyte file you can’t open. Centralizing can start simple: rsyslog or a lightweight agent shipping logs to a dedicated machine or a cloud logging service.
A one-person team doesn’t need a heavyweight stack like ELK on day one. Centralized storage plus a search interface is enough to start; consider more once volume grows. If you run a CDN site, collect both edge logs and origin logs. See CDN global acceleration.
| Stage | How to do it | Tool / Config | What happens if you skip it |
|---|---|---|---|
| Write the file | Consistent output format | combined format | Missing fields make analysis scripts painful to write |
| Rotation | Daily rotation with compression | logrotate | Oversized files fill up the disk |
| Centralization | Aggregate to one store | rsyslog / cloud logging | History is lost when you switch machines |
| Alerts | Error-rate threshold alerts | Monitoring platform rules | You only find out when users complain |
3. Error logs: the one a one-person team should watch closest
Of the three log types, error logs carry the most information per line and deserve your attention first. Access logs run to hundreds of thousands of lines a day; error logs may be only a few hundred, but every line is a real failure. Centralizing error logs separately and alerting on them separately gives you the best return on effort.
Set thresholds as concrete numbers: alert immediately when the 5xx error rate exceeds 1% for 10 straight minutes; alert on a single abnormal error (such as a database connection failure or a failed payment callback) the moment it appears, without waiting for it to accumulate. Grade your error logs: keep warnings for periodic review, but errors and criticals must notify in real time. For apps like PHP, also pull out fatal errors and slow queries separately.
Example scenario: One morning the alert fires: the 5xx error rate jumps from 0.1% to 3%. You open the error logs first and see a flood of “PHP-FPM: unable to connect”, which points to a full backend process pool. Then you check access logs to see which URLs triggered it and find a newly launched endpoint whose slow query is dragging down the pool. Finally you check application logs to confirm which SQL it was. You pinpoint it in ten minutes. That’s the value of starting with error logs.
| Error level | Example | What to do | Alert threshold |
|---|---|---|---|
| critical | Database down, payment failed | Notify immediately | Alert on first occurrence |
| error | 5xx, fatal errors | Real-time notification | > 1% within 10 minutes |
| warning | Slow queries, retries | Review in daily summary | Summarize when 2x above baseline |
| notice | Normal business notices | Archive, no alert | No alert |
4. Application logs: make business flows traceable
Access logs and error logs are the server’s “objective” record, but they can’t answer business questions like “why didn’t this inquiry reach the CRM?” That’s what application logs are for: you write logs at key business points yourself, such as inquiry submitted, written to the database, pushed to the CRM, CRM callback failed.
The key to application logs is that every entry carries a business identifier, such as an inquiry ID or order number. That way, when something goes wrong, you search that ID and pull up the full trail of that business from submission to completion. Don’t write meaningless logs (“executed line 10”); every entry should answer a business question. Keep log volume under a few dozen MB per day, and don’t dump entire request bodies into the log, which is both slow and leaks private data.
The link between slow performance and logging is also worth mentioning: too much logging, or writing to disk synchronously on every request, will itself slow down responses. For how to read slow-response metrics, see Core Web Vitals. For the overall monitoring and alerting framework, see Site monitoring and alerts.
| Business step | What to record | Identifier carried | How to investigate when something goes wrong |
|---|---|---|---|
| Inquiry submitted | Form fields, source page | Inquiry ID | Search submission records by ID |
| Saved to database | Write succeeded / failed | Inquiry ID | Check whether it was persisted |
| Pushed to CRM | API return code, duration | Inquiry ID | Check whether CRM received it |
| Email notification | Send succeeded / failed | Inquiry ID | Check whether sales received it |
5. Pitfalls: the three most common logging mistakes
The first pitfall is “no rotation, no retention.” Logs keep writing to a single file, and three months later that file is several GB, so a grep during troubleshooting hangs for ages, or it fills the disk and takes your site down. Daily rotation plus a retention policy is the baseline.
The second pitfall is “keep only real-time, never archive.” When the disk fills up, the oldest data gets deleted, so when you want to trace the start of an incident from two months ago, there’s nothing there. Keep hot data for 90 days and compress and archive older data for at least a year; the cost is very low. The third pitfall is “writing sensitive information into logs.” Putting user passwords or full credit card numbers into logs is like building another leak source inside your log store. Mask sensitive fields or keep them off disk entirely.
| Common mistake | What it looks like | Real problem | Fix |
|---|---|---|---|
| No rotation | Single file keeps growing | Disk fills up, searches slow | Rotate daily with logrotate |
| No history kept | Delete once disk is full | No data when you need to look back | Hot 90 days, archive 1 year |
| Sensitive info written | Logs make debugging easy | Log store becomes a leak source | Mask sensitive fields |
| Three types mixed together | Dig through each one when something breaks | Slow to locate, noisy | Errors first, search by category |
FAQ
Does an independent site really need a logging platform like ELK?
Why can’t access, error, and application logs be stored together?
How long should each type of log be kept?
How should error log alert thresholds be set?
How detailed should application logs be, and will that hurt performance?
Can user passwords and payment information be recorded in logs?
Final thoughts
The value of log management isn’t in “how much you store” but in “can you pinpoint the problem within three minutes when something goes wrong.” For a solo-run independent site, separate the three log types, rotate and centralize them daily, and alert when the error rate exceeds 1%. Once this foundation is in place, daily troubleshooting time drops from “digging through logs for half a day” to “glancing at an alert.” Heavyweight log platforms are for after you’ve scaled up, not a starting requirement.
Three steps to get this live: first, store access, error, and application logs separately, rotate them daily, and set a retention period for each. Second, centralize them with rsyslog or a cloud logging service so you can search by business ID. Third, alert when the error rate exceeds 1% over 10 minutes, or the moment a critical error appears. Do these three things and your view of production shifts from “I only find out when users complain” to “I know before they do.” For the ongoing maintenance rhythm, see Website Maintenance Schedule and plan them together.
One line worth remembering: logs aren’t old files you keep for nostalgia, they’re the dashcam at the scene of the accident. If you never sort or centralize them, all you can do when something breaks is guess from memory.